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PREFACE 


The  Institute  for  Defense  Analyses  (IDA)  prepared  this  paper  for  the  Defense 
Threat  Reduction  Agency  (DTRA),  in  partial  fulfillment  of  the  task  “Support  for  DTRA 
in  the  Validation  Analysis  of  Hazardous  Material  Transport  and  Dispersion  Prediction 
Models.”  The  objective  of  this  effort  was  to  conduct  analyses  and  special  studies 
associated  with  the  verification,  validation,  and  accreditation  (VV&A)  of  hazardous 
transport  and  dispersion  prediction  models.  This  task  involves  the  comparison  of  two 
models:  DTRA’s  Hazard  Prediction  and  Assessment  Capability  (HP AC)  and  Lawrence 
Livermore  National  Laboratory’s  (LLNL)  National  Atmospheric  Release  Advisory 
Center  (NARAC)  modeling  system.  An  additional  component  of  this  task,  addressed  in 
this  report,  is  the  exploration  and  development  of  validation  and  accreditation  measures 
of  effectiveness  for  transport  and  dispersion  models. 

The  IDA  Technical  Review  Committee  was  chaired  by  Thomas  P.  Christie  and 
consisted  of  John  N.  Bombardt  Jr.,  Arthur  Fries,  Jeffrey  H.  Grotte,  David  A.  McWhorter, 
and  Jozsef  A.  Toth.  The  authors  would  like  to  thank  Allan  Reiter  (DTRA),  Leon  Wittwer 
(DTRA),  and  Don  Ermak  (LLNL)  for  their  comments,  critiques,  and  support  throughout 
this  effort. 

The  LLNL  work  was  performed  under  the  auspices  of  the  U.S.  Department  of 
Energy  by  University  of  California  Lawrence  Livermore  National  Laboratory  under 
contract  No.  W-7405-Eng-48. 
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SUMMARY 


A.  BACKGROUND 

In  1999,  IDA  conducted  a  study  for  the  Office  of  the  Secretary  of  Defense  that,  in 
part,  explored  military  user  requirements  for  hazard  predictions.1  During  that  study,  it 
became  apparent  that  there  was  a  need  for  measures  that  clearly  communicate  to  the  user 
the  relative  worth  of  a  model’s  predictions. 

During  fiscal  year  2000,  a  series  of  studies  in  support  of  the  Defense  Threat 
Reduction  Agency  (DTRA)  was  begun.  The  goal  of  these  studies  is  to  improve  the 
verification,  validation,  and  accreditation  (VV&A)  of  hazard  prediction  and  assessment 
models  and  capabilities.  These  studies  are  part  of  a  larger  joint  VV&A  effort  that  DTRA 
and  the  Department  of  Energy,  via  the  Lawrence  Livermore  National  Laboratory 
(LLNL),  are  conducting.  This  joint  effort  includes  comparisons  of  the  LLNL  and  DTRA 
transport  and  dispersion  (T&D)  modeling  systems,  NARAC  and  HPAC,  respectively.2 

B.  PURPOSE 

The  purpose  of  this  paper  is  to  introduce  and  describe  a  family  of  novel  user- 
oriented  MOEs,  focusing  on  their  interpretation  and  usage.  The  application  of  these 
MOEs  to  the  comparison  of  HPAC  and  NARAC  predictions  of  the  1956  Prairie  Grass 
field  trial  observations  is  also  described  in  this  paper.3 

C.  USER-ORIENTED  MEASURES  OF  EFFECTIVENESS  (MOEs) 

There  is  an  enduring  user  need  for  measures  that  meaningfully  communicate  the 
relative  worth  (and  validity)  of  a  model’s  predictions  when  applied  in  a  given  regime 


1  NBC  Hazard  Prediction  Model  Capability  Analysis,  IDA  Document  D-2245,  September  1999. 

2  NARAC  =  National  Atmospheric  Release  Advisory  Center  and  HPAC  =  Hazard  Prediction  and 
Assessment  Capability. 

3  The  1956  Prairie  Grass  field  trials  represent  one  of  the  best-characterized  short-range  transport  and 
dispersion  experiments.  In  addition,  these  trials  correspond  to  a  baseline  against  which  many  transport 
and  dispersion  models  have  been  compared.  Therefore,  these  field  trials  were  chosen  for  this  initial 
study.  Additional  descriptions  of  the  Prairie  Grass  field  trials  can  be  found  in  Chapter  1,  Section  B 
and  Chapter  2,  Section  C. 
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(e.g.,  short-range  chemical  agent  or  longer-range  biological  agent,  unstable  or  stable 
meteorological  conditions).  For  a  user-oriented  measure,  we  desire  the  following 
properties: 

.  Clear-cut  interpretation  for  consistent  utility  and  relatively  widespread 
communication  potential 

.  Sensitivity  to  real  changes  in  model  performance  (e.g.,  can  detect  differences 
between  various  models) 

.  Robustness  with  respect  to  small  measurement  errors  and  uncertainties. 

The  key  objective  for  an  interpretive  user-oriented  MOE  is  its  ability  to 
communicate  simply  (and  appropriately)  the  relative  performance  of  a  model  in  a  given 
application.  It  is  important  that  other  measures,  for  instance,  the  other  statistical 
measures  described  later  in  this  paper,  also  be  examined  for  any  set  of  data  that  is  to  be 
described.  In  order  for  such  a  user-oriented  MOE  to  have  widespread  credibility,  the 
conclusions  communicated  by  the  MOE  should  be  consistent  with  the  conclusions  one 
would  obtain  from  a  detailed  review  of  other  statistical  measures  and  graphical 
representations.  Where  there  are  inconsistencies,  the  differences  should  be  highlighted 
and  understood. 

A  fundamental  feature  of  any  comparison  of  hazard  prediction  model  output  to 
observations  is  the  over-  and  under-prediction  regions.  We  define  false  negative  where  a 
hazard  is  observed  but  not  predicted,  and  false  positive  where  a  hazard  is  predicted  but 
not  observed.  Figure  1  shows  the  observed  and  predicted  area  at  the  same  dosage  level 
for  some  nominal  situation.  Numerical  values  in  three-dimensional  space  characterize 
this  conceptual  view.  These  values  are  associated  with  estimates  of  the  false  negative 
region  (Arm),  the  false  positive  region  (AFP),  and  the  overlap  region  (Aov). 
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The  first  measure  that  we  describe,  MOE  1,  combines  the  area  of  overlap,  the  area 
of  false  negative,  and  the  area  of  false  positive  to  form  a  single  value  as  shown  in 
equation  1. 

MOE  1  =  7 - ^ - r.  (1) 

\AoV  CpN^-FN  +  CppApp  ) 

In  this  equation,  Cfn  and  Cfp  represent  user  coefficients  that  weight  the  relative 
importance  of  the  false  positive  and  false  negative  areas.  The  coefficient  values  chosen 
should  be  related  to  the  specific  model  application  being  examined  and  the  particular 
user’s  risk  tolerance.  If  we  consider  Cfn  =  Cfp  =  1,  that  is,  equal  weights  for  false 
positive  and  false  negative  areas,  then  this  MOE  simply  is  the  fraction  of  the  total  area 
that  constitutes  overlap  between  the  model  and  the  prediction.  A  perfect  model  would 
have  an  MOE  value  of  1  (i.e.,  all  overlap)  and  a  perfectly  terrible  model  would  have  an 
MOE  value  of  0  (i.e.,  no  overlap). 


The  second  measure  that  we  consider,  MOE  2,  has  two  dimensions.  The  x-axis 
corresponds  to  the  ratio  of  overlap  area  to  observed  area  and  the  y-axis  corresponds  to  the 
ratio  of  overlap  area  to  predicted  area.  These  mathematical  definitions  can  be 
algebraically  rearranged  (equation  2  below)  and  we  then  recognize  that  the  x-axis 
corresponds  to  1  minus  the  false  negative  fraction  and  the  y-axis  corresponds  to  1  minus 
the  false  positive  fraction. 


MOE  2  = 


where  Afn  =  area  of  false  negative,  App  =  area  of  false  positive,  Aov  =  area  of  overlap, 
Apr  =  area  of  the  prediction,  and  Aob  =  area  of  the  observation.  Consistent  with  the 
above  algebraic  rearrangement,  Figure  2  shows  the  area  of  false  negative  decreasing  from 
left  to  right. 


We  imagine  associating  a  given  user  and  application  with  specific  risk  tolerances, 
ri  and  xi,  therefore  accepting  from  a  given  prediction  a  certain  false  positive  and  false 
negative  fraction.  These  risk  tolerances  define  in  the  MOE  space  an  acceptable  or 
desirable  region  for  model  performance. 
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Afn  decreases  — ► 

Better  — ► 

Figure  2.  Two-Dimensional  MOE:  MOE  2 

The  various  areas  described  in  equations  1  and  2  can  be  estimated  from  field  trial 
observations  and  model  predictions.  For  example,  we  considered  the  distance  traversed 
by  samplers  along  an  arc  at  which  both  the  observations  and  predictions  led  to  dosages 
above  some  selected  threshold,  as  a  natural  estimate  of  the  overlap  “area.”  An  analogous 
methodology  was  used  to  estimate  Afn  and  AFP.  We  denote  estimates  determined  in  this 
way  with  the  label  “AE1.”  The  AE1  estimate  represents  a  natural  analog  to  a  hazard  area 
at  a  given  critical  threshold.  However,  it  should  be  noted  that  a  perfect  MOE  score  based 
on  AE1  does  not  imply  perfect  agreement  between  the  observed  and  predicted  values  at 
all  measurement  locations,  since  only  the  points  above  a  particular  threshold  dosage 
value  are  considered.  Figure  3  illustrates  this  procedure. 

Next,  we  considered  the  summed  dosages.  For  example,  for  AFP,  all  of  the 
samplers  with  predictions  of  greater  value  than  the  observations  can  be  considered.  One 
then  sums  the  differences  between  the  predicted  and  observed  dosages  at  those  samplers 
to  generate  AFP.  Based  on  the  samplers  that  contained  observed  values  that  were  larger 
than  the  predicted  values ,  one  can  similarly  compute  Arm.  Aov  is  calculated  by 
considering  all  samplers  and  summing  the  dosages  associated  with  the  minimum 
predicted  or  observed  value.  We  refer  to  this  procedure  as  AE2  and  it  is  illustrated  in 
Figure  4  4 


4  Theoretically,  the  computation  of  AE2 -based  values  does  not  require  the  use  of  a  threshold.  However, 
field  trial  data  will  always  have  uncertainties  associated  with  the  smallest  observations  -  that  is, 
experimental  resolution.  Therefore,  AE2 -based  computations  that  use  field  trial  data  require  a  defined 
threshold. 
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Figure  4.  Illustration  of  AE2  Procedure 
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AE2  is  based  on  comparison  of  observed  and  predicted  values  at  every  point  and 
is  a  measure  of  under-  and  over-prediction.  A  perfect  MOE  score  based  on  AE2  implies 
an  exact  match  between  observed  and  predicted  values.  We  consider  AE2-based  MOEs 
to  correspond  to  validation  measures  that  can  have  diagnostic  potential.  In  contrast,  an 
AE1 -based  MOE,  if  it  includes  a  user-defined  critical  threshold,  can  serve  as  an 
operational  or  accreditation  measure. 

We  also  interpreted  the  area  estimates  (i.e.,  Aov,  Afn,  and  App)  in  terms  of  a  filter 
that  considered  the  effects  of  a  presumably  toxic  agent  on  an  exposed  population.5  First, 
the  lethality  or  effects  of  the  agent  being  studied  and  the  dosages  at  a  given  sampler  are 
examined.  For  example,  standard  probit  curves  might  be  used  to  assess  the 
lethality/effects  of  exposure  at  a  given  sampler.  Next,  the  dosages  are  converted  into  a 
fractional  lethal/effects  exposure,  that  is,  the  fraction  of  an  exposed  population,  at  that 
dosage  level,  expected  to  become  a  casualty.6  For  example,  if  the  observed  or  predicted 
dosage  were  LD50  (lethal  dosage  for  50  percent  of  the  exposed  population),  then  half  of 
the  exposed  population  would  be  expected  to  die. 

Figure  5  provides  a  nominal  illustration  of  this  calculation.  First,  the  observed 
dosage  at  sampler  17  is  “10  FD5os”  (10  x  FD50)  and  the  predicted  dosage  (by  HP  AC  in 
this  illustration)  is  1,000  FD50S.  These  differences  can  be  converted  into  the  fraction  of 
the  exposed  population  that  would  be  expected  to  become  a  casualty.  Assume  that  for  the 
probit  curve  associated  with  the  hypothetical  agent,  a  dosage  of  10  x  FD50  means  that  90 
percent  of  the  population  is  exposed  to  a  lethal  dosage,  while  a  dosage  of  1,000  x  LD50 
implies  99.99  percent  of  the  population  accumulates  a  lethal  dosage.  For  the  overlap 
area,  Aov  -  that  is,  10  FD50S  -  0.90  of  the  exposed  population  would  be  expected  to 
accumulate  a  lethal  dosage.  At  this  sampler,  the  model  overpredicts  the  observation  and 
the  area  of  false  positive,  AFp,  corresponds  to  an  additional  0.10  lethal  exposure 
population  fraction.  Then  for  each  trial  (or  arc  or  sampler),  estimates  of  Aov,  Afn,  and 
App  can  be  obtained  in  this  way.7 


5  As  applied  here,  we  assumed  a  spatially  uniform  exposed  population. 

6  Determining  actual  casualty  levels  for  a  specific  scenario  would  also  need  to  include  factors  such  as 
the  means  of  exposure  (respiratory  or  percutaneous)  and  other  parameters  relating  to  that  exposure 
(e.g.,  breathing  rate  and  the  effect  of  clothing). 

7  Chapter  2,  Section  B.l.f  provides  additional  details. 
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Figure  5.  Nominal  Lethality/Effects  Filter  Illustrated 

By  accounting  for  the  lethality/effects  of  a  given  hazard,  this  measure 
operationally  weights  the  over-  or  under-prediction  of  a  model  (with  respect  to  a  set  of 
observations).  Therefore,  differences  between  observations  and  predictions  that  would 
lead  to  additional  deaths  or  casualties  are  considered  more  important  than  those  that, 
perhaps,  lead  to  very  different  dosages,  but  a  similar  fraction  of  lethal/effects  exposure. 
In  a  sense,  the  above  process  filters  our  interpretation  of  the  above  areas  (i.e.,  A0v,  AFP, 
and  Afn)  through  the  appropriate  lethality/effects  “lens.”  Hence,  we  refer  to  MOE  results 
obtained  in  this  way  as  “lethality/effects-filtered.” 

Perhaps  the  key  feature  of  the  MOEs  described  in  this  paper  is  their  inclusion  of 
directional  effects.  That  is,  in  order  to  “score”  well  in  terms  of  these  MOEs,  the  hazard 
prediction  modeling  system  must  predict  the  shape  and  location  (e.g.,  direction)  of  the 
plume  accurately.  One  advantage  of  these  MOEs  is  that  mean  value  and  probabilistic 
outputs  can  be  directly  compared.  Therefore,  a  modeling  system  that  creates  a 
probabilistic  plume,  that  is  meant  to  encompass  many  possible  plume  realizations,  can  be 
scored  with  this  MOE.8 


In  addition  to  the  two  MOEs  described  above,  eight  other  statistical  measures 
were  computed  for  the  model  comparisons  described  in  this  paper.  Two 
observed/predicted  values  were  considered  for  the  calculation  of  each  of  these  statistical 


8  An  upcoming  IDA  paper  describes  the  results  of  the  application  of  MOEs  to  probabilistic  outputs 
generated  by  HPAC. 
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measures.  First,  for  each  arc  the  maximum  dosage  was  considered  -  ArcMax.  Next,  the 
crosswind-integrated  dosage  (CWI)  was  examined.  The  statistical  measures  computed 
include  fractional  bias  (FB),  geometric  mean  bias  (MG),  normalized  mean  square  error 
(NMSE),  geometric  mean  variance  (VG),  correlation  coefficient  (R),  fraction  of 
predictions  within  a  factor  of  2  (FAC2),  fraction  of  predictions  within  a  factor  of  5 
(FAC5),  and  fraction  of  predictions  within  a  factor  of  10  (FAC  10). 9 

D.  CONCLUSIONS  BASED  ON  THE  EXAMINATION  OF  MOE  ESTIMATES10 

The  two-dimensional  (2D)  measure  (MOE  2),  with  false  positive  and  false 
negative  fractions  considered  orthogonal,  consistently  resolved  important  model 
performance  features.  MOE  2  was  useful  in  identifying  and  describing  differences 
between  HPAC  and  NARAC  predictions  of  the  Prairie  Grass  trials. 1 1 

Figure  6  compares  MOE  2  estimates  for  HPAC  and  NARAC  predictions.  The 
x-axis  corresponds  to  decreasing  false  negative  fraction,  from  left  to  right,  and  the  y-axis 
corresponds  to  decreasing  false  positive  fraction,  from  bottom  to  top.12  Each  colored 
area  in  the  figure  represents  the  estimated  95th  percent  confidence  region  for  the  given 
MOE  2  point  estimate.13  The  complete  separation  of  these  two  regions  implies  that  the 
differences  between  the  HPAC  and  NARAC  MOE  2  point  estimates  are  statistically 
significant. 


9  In  addition,  NMSE  and  FB  were  computed  for  an  estimate  of  the  plume  width  and  the  root  mean 
square  error  (RMSE)  and  bias  were  computed  for  the  centerline  direction.  Appendix  J  provides 
additional  details. 

1(1  The  Prairie  Grass  simulations  used  for  this  study  were  performed  using  a  protocol  designed  to  allow 
analysis  of  differences  in  dispersion  model  physics.  As  much  as  possible,  meteorology,  resolution,  and 
user  skill  level  differences  were  minimized.  As  such,  the  results  identified  interesting  differences  in 
the  models,  but  should  not  be  viewed  as  a  conclusive  comparison  of  the  HPAC-NARAC  results  in 
terms  of  their  best  possible  performance  against  the  Prairie  Grass  data  set. 

11  This  section  illustrates  the  main  results  and  conclusions  of  this  analysis  based  on  MOE  2.  MOE  1 
results  were  similar  and  consistent,  albeit  not  always  as  informative.  Chapter  3,  Section  A.l  and 
Appendix  H  provide  detailed  MOE  1  results  and  comparisons. 

12  As  described  in  equation  2,  1-Afn/A0b  (x-axis)  and  1-AFP/APR  (y-axis)  can  be  algebraically  rearranged 
to  give  Aov/Aob  and  A0v/APR,  respectively.  Throughout  this  paper  -  the  main  body  and  appendix  H  - 
we  typically  refer  to  the  x-  and  y-axis  as  decreasing  false  negative  (i.e.,  1-Afn/A0b)  and  decreasing 
false  positive  (1-AFP/APR),  respectively.  This  labeling  scheme  tends  to  emphasize  the  false  positive 
and  false  negative  aspects  of  the  MOE. 

13  These  95th  percent  confidence  regions  are  based  on  1,000  bootstrap  samples.  See  Chapter  2,  Section 
B.3  for  additional  details. 
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The  HP  AC  MOE  2  AE1  threshold  area  estimate  (0.80,  0.85)  lies  somewhat  closer 
than  the  corresponding  NARAC  estimate  (0.94,  0.62)  to  the  best  possible  value  of  (1,1). 
The  AE2  estimates,  in  which  the  “areas”  are  based  on  the  summed  crosswind  dosages, 
show  similar  results  albeit  with  the  NARAC  estimate  (0.74,  0.71)  closer  to  (1,1)  than  the 
HP  AC  estimate  (0.60,  0.79).  Figure  6  indicates  that  the  NARAC  predictions  exhibit  a 
smaller  false  negative  fraction  relative  to  the  HP  AC  predictions,  at  least  on  average. 
Similarly,  the  HP  AC  predictions  show  smaller  false  positive  fractions  relative  to  the 
corresponding  NARAC  predictions.  One  reasonable  implication  is  that  the  NARAC- 
predicted  plumes  are,  on  average,  wider  than  the  HP  AC -predicted  plumes,  at  least  as  the 
models  were  employed  in  this  study.  This  was  verified  by  examining  graphical 
representations  of  the  predicted  and  observed  plumes  and  by  computing  standard 
statistical  measures  (as  described  in  the  main  body  and  appendices). 


Figure  6.  MOE  2  Comparisons  of  HPAC  and  NARAC  Predictions  of  51  Prairie  Grass  Trials 

Variances  in  a  given  model’s  prediction  performance  as  a  function  of  arc  range 
and  stability  condition  were  easily  detected  and  typically  led  to  statistically  significant 
conclusions.  Figure  7  presents  confidence  region  estimates  for  MOE  2  as  a  function  of 
meteorological  stability  category  grouping  (SCG).  For  this  figure,  three  independent 
meteorological  stability  category  groupings  were  examined:  unstable  to  very  unstable 
(SCG  =  1,2);  near  neutral  to  somewhat  stable  (SCG  =  3,4,5);  and  stable  to  very  stable 
(SCG  =  6,7).  The  same  51  trials  that  were  described  in  Figure  6  are  included  in  Figure  7 
albeit  categorized  by  SCG. 
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Figure  7.  MOE  2  Comparisons  of  HPAC  and  NARAC  Predictions  of  51  Prairie  Grass  Trials 
as  a  Function  of  Stability  Category  Grouping 

With  respect  to  HPAC  and  NARAC  comparisons,  the  MOE  2  (AE1  and  AE2) 
results  are  most  similar  for  the  trials  conducted  during  the  more  unstable  conditions  (SCG 
=  1,2)  with  the  NARAC  predictions  generating  slightly  larger  false  positive  fractions  and 
slightly  smaller  false  negative  fractions.  This  trend  is  magnified  for  the  near- neutral  trials 
(SCG  =  3,4,5).  The  biggest  separation  is  associated  with  the  MOE  2  AE1  estimates  for 
the  stable  trials  (SCG  =  6,7). 14  For  both  the  neutral  and  the  stable  conditions,  the 
NARAC  predictions  generate  relatively  little  area  (AE1)  associated  with  false  negatives 
(i.e.,  along  the  x-axis  the  confidence  region  lies  close  to  1).  The  HPAC  predictions  of  the 
neutral  and  stable  trials  lead  to  larger  false  negative  but  smaller  false  positive  fractions 
(AE1). 

The  MOE  2  AE2  results  (summed  dosage-based  “areas”)  for  stable  trials,  indicate 
decreased  false  negative  fractions  associated  with  NARAC  predictions  relative  to  HPAC 
predictions  and  similar  false  positive  fractions  for  both  sets  of  predictions.  However,  the 
HPAC  predictions  of  the  stable  trials  (SCG  =  6,7)  show  a  wider  range  of  false  negative 
and  false  positive  values  (relative  to  the  NARAC  predictions).  This  can  be  seen  in  the 


II * * 4  A  separate  independent  grouping  of  stability  categories  (SCGs  =  1,2,3;  4;  and  5,6,7)  was  also 

examined  with  similar  results.  See  Chapter  3  and  Appendix  H.  A  more  physically-based  arrangement 

would  be  to  group  the  near-neutral  categories,  3  and  4,  together.  The  conclusions  of  this  paper  were 

not  substantially  affected  by  this  regrouping. 
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strong  directionality  (i.e.,  elliptical  eccentricity)  associated  with  the  stable  trials’ 
confidence  region  for  the  HP  AC  prediction.  This  result  is  consistent  with  a  few  narrow 
plume  predictions  and  observations  being  slightly  offset  in  direction.  Therefore,  for  those 
few  trials,  both  false  positive  and  false  negative  fractions  are  increased  at  the  expense  of 
the  overlap  fraction.15 

Both  HP  AC  and  NARAC  appear  to  perform  the  best,  with  respect  to  being  closest 
to  the  best  case  (1,1)  value,  for  the  relatively  unstable  trials.  Similarly,  both  models 
performed  worst,  as  measured  by  MOE  2,  for  the  trials  that  were  conducted  under  the 
more  stable  conditions.16 

Based  on  relatively  small  changes  in  HPAC  inputs,  we  were  also  able  to  discern 
prediction  performance  differences  (between  HPAC  runs)  after  examining  the  computed 
MOE  2  estimates.  These  changes  in  HPAC  inputs  included  adding  a  mechanism  for  SO2 
deposition  and,  separately,  modifying  a  default  value  associated  with  dispersion 
assumptions  during  the  lightest  wind  conditions.  Similarly,  changes  to  NARAC  inputs 
(i.e.,  the  inclusion  of  a  surface  deposition  mechanism  and  changes  to  a  dispersion 
parameter  -  sigma-v)  led  to  detectable  differences  in  MOE  2  results.  Furthermore,  we 
found  that  the  2D  MOE  (MOE  2)  allowed  for  straightforward  communication  of  the 
impact  of  these  modest  input  changes. 

We  found  that  the  lethality/effects  filter  can  be  used  to  compute  MOE  values,  and 
can  be  characterized  as  a  “tunable  dial”  that  can  relate  the  goodness  of  a  prediction  for 
agents  of  greatly  varying  toxicity  (e.g.,  biological  versus  chemical  agents  or  threshold 
versus  lethal  effects  for  a  chemical  agent).  This  feature  may  make  this  methodology  of 
particular  value  with  respect  to  user  accreditation.  For  instance,  the  specific  application 
and  user  will  dictate  the  agent  type,  effect,  and  downwind  distance  of  interest,  therefore 
defining  how  the  lethality/effects  dial  should  be  tuned. 

Overall  the  2D  MOE  revealed  the  following  with  respect  to  comparisons  of 
HPAC  and  NARAC  predictions  under  the  specified  model  intercomparison  protocol: 


15  For  MOE  2  AE2,  using  the  default  uu(calm)  input  value,  vice  the  baseline  uu(calm)  value,  improved 
the  HPAC  stable  trials’  point  estimate  and  greatly  reduced  the  associated  elliptical  eccentricity  and 
confidence  region  size. 

16  For  these  data,  we  found  that  the  conclusions  based  on  the  standard  statistical  measures  were  consistent 
with  those  based  only  on  the  MOEs.  Chapter  3,  Section  B  provides  a  few  additional  comments  on  the 
usage  and  value  of  these  standard  statistical  measures. 
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.  HPAC  and  NARAC  predictions  of  the  relatively  unstable  Prairie  Grass  trials 
were  most  similar.  Predictions  of  the  most  stable  trials  led  to  the  biggest 
differences. 

.  In  a  sense,  the  NARAC  predictions,  as  run,  were  more  conservative  than  the 
HPAC  predictions.  In  general,  NARAC  predictions  led  to  substantially 
smaller  false  negative  fractions,  but  at  the  expense  of  larger  false  positive 
fractions.  This  was  most  true  for  the  predictions  of  the  neutral  and  stable 
trials.  In  contrast,  HPAC  predictions  showed  substantially  smaller  false 
positive  areas,  but  at  the  expense  of  larger  false  negative  areas.  A  review  of 
Appendix  E  confirms  this  interpretation. 

This  study  also  used  a  variety  of  other  statistical  and  graphical  measures  of  model 
performance.  Examination  of  plots  of  measured  and  predicted  dosages  as  a  function  of 
distance  helped  support  conclusions.  Statistical  measures  for  derived  quantities  -  such  as 
ArcMax,  CWI  dosage,  plume  horizontal  spread,  and  mean  plume  direction  -  were  of 
some  value  in  confirming  the  underlying  reasons  for  the  MOE  results.  MOE  results  (for 
both  AE1  and  AE2)  include  the  effects  of  directional  error  by  pairing  observed  and 
predicted  dosage  in  space  (and  time).  Separate  comparisons  of  ArcMax,  CWI  dosage, 
plume  spread,  and  mean  plume  direction  (which  do  not  measure  mean  wind  direction 
error)  can  allow  for  the  isolation  of  errors  due  to  the  dispersion  modeling  methods  from 
those  due  to  mean  wind  advection  modeling  errors  (particularly  directional).  The  results 
of  this  study  show  that  these  MOEs  are  a  potentially  valuable  tool,  but  that  no  single 
measure  can  be  used  exclusively  in  evaluating  and  analyzing  model  performance. 

E.  OUTLINE  OF  THIS  PAPER 

This  paper  is  divided  into  three  chapters  and  fourteen  appendices.  Chapter  1 
provides  a  brief  introduction  and  Chapter  2  describes  this  study’s  methodologies. 
Chapter  2  defines  and  develops  two  user-oriented  MOEs,  MOE  1  and  MOE  2,  and 
discusses  a  few  statistical  measures  that  have  often  been  used  for  the  evaluation  of 
transport  and  dispersion  models.  Chapter  3  provides  the  results  of  the  application  of 
these  MOEs  to  the  comparison  of  HPAC  and  NARAC  predictions  of  the  Prairie  Grass 
field  trials. 

Appendix  A  provides  an  acronym  list  and  Appendix  B  presents  some  details 
associated  with  the  model  predictions  that  were  used  for  this  study’s  comparisons. 
Appendices  C,  D,  E,  F,  G,  L,  and  M  present  comparisons,  by  sampler  arc,  of  Prairie 
Grass  field  trial  observations  and  model  predictions.  Detailed  model  comparison  results, 
using  MOEs  and  standard  statistical  measures,  are  deposited  in  Appendices  H  and  I, 
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respectively.  Appendix  J  briefly  describes  some  details  of  our  methodology,  and  possible 
alternatives,  for  estimating  a  plume  width  and  direction  (associated  with  a  given  arc)  and 
Appendix  K  describes  a  few  corrections  that  were  applied  to  the  field  trial  data  set  that 
we  used.  Finally,  Appendix  N  includes  an  extract  from  the  pertinent  task  order. 
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CHAPTER  1 

INTRODUCTION 


1.  INTRODUCTION 


A  series  of  studies  in  support  of  the  Defense  Threat  Reduction  Agency  (DTRA) 
was  begun  in  Fiscal  Year  2000.  The  goal  of  these  studies  is  to  improve  the  verification, 
validation,  and  accreditation  (VV&A)  of  hazard  prediction  and  assessment  models  and 
capabilities  (e.g.,  HPAC  and  NARAC)1.  These  studies  are  part  of  a  larger  joint  VV&A 
effort  that  DTRA  and  the  Department  of  Energy  (DOE),  via  the  Lawrence  Livermore 
National  Laboratory  (LLNL),  are  conducting.  This  joint  effort  includes  comparisons  of 
the  LLNL  and  DTRA  transport  and  dispersion  (T&D)  modeling  systems,  NARAC  and 
HPAC,  respectively,  and  their  predictions.  IDA’s  role  is  to  conduct  independent  analysis 
and  special  studies  associated  with  this  VV&A  effort.  This  role  includes  conducting 
comparisons  between  the  models,  providing  analysis  and  discussions  associated  with 
these  examinations,  and  exploring  and  developing  measures  of  effectiveness  (MOE)  that 
can  aid  hazard  prediction  model  validation  and  accreditation.2 

This  paper  develops  novel  user-oriented  MOEs  and  applies  them  to  the 
comparison  of  two  hazard  prediction  models.  The  usage  and  interpretation  of  these 
MOEs  and  their  resulting  values  are  described  for  short-range  field  trial  data.  In  addition, 
“standard”  statistical  measures  are  computed  and  some  comparisons  to  the  proposed 
MOEs  are  provided. 

A.  BACKGROUND 

We  refer  to  this  joint  VV&A  effort  as  a  cooperative  comparison.  That  is,  DTRA 
and  LLNL  have  set  up  a  funded  team  that  works  closely  together  to  provide  comparable 
model  runs  and  analysis.  The  two  models  have,  over  the  years,  been  developed 
independently  and  against  different  requirements.  Lurther,  we  speculate  that  it  is  more 
efficient  for  the  various  agencies  to  learn  about  other  government  models  and  tools  in  a 
cooperative  (joint  effort)  manner.  Recent  newsletter  articles  have  suggested  the  value 
and  challenges  of  hazard  prediction  model  VV&A  [Ref.  1-1]  and  concluded  that  joint 


1  NARAC  =  National  Atmospheric  Release  Advisory  Center  and  HPAC  =  Hazard  Prediction  and 
Assessment  Capability. 

2  Appendix  N  of  this  document  contains  an  extract  from  the  pertinent  Fiscal  Year  2000  task  order. 
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efforts  in  the  Chemical  and  Biological  (CB)  warfare  modeling  and  simulation  arena  are 
expected  to  bring  a  certain  synergy  to  the  community  [Ref.  1-2]. 

Future  studies  are  expected  to  include  comparisons  of  model  algorithms,  VV&A 
efforts,  and  reporting  methods.  Careful  comparisons  to  relatively  simple,  previously 
collected,  field  trial  data  have  been  completed.  In  addition,  comparisons  of  model 
outputs  over  a  small  set  of  controlled  input  conditions  have  also  been  completed.  We 
refer  to  this  first  set  of  comparisons  as  model-to-field  trial  comparisons  (or  simply  MFT). 
The  second  set  of  examinations  is  referred  to  as  model-to-model-only  comparisons 
(MMO).3  This  paper  documents  some  of  the  methodology,  results,  and  analyses 
associated  with  the  first  in  a  series  of  MFT  studies,  with  an  emphasis  on  the  presentation 
and  application  of  user-oriented  MOEs.  The  field  trial  that  was  selected  for  this 
comparison  is  the  1956  Prairie  Grass  experiment  [Ref.  1-4]  described  briefly  later  in  this 
chapter. 

It  is  anticipated  that  even  these  simple  MFT  and  MMO  comparisons  should  lead 
to  the  identification  of  differences  in  model  behavior  and  the  quantification  of  the 
potential  magnitude  of  such  differences  under  a  variety  of  conditions.4  Importantly,  these 
initial  studies  are  expected  to  serve  as  the  basis  for  future  more  complex  comparisons 
(e.g.,  more  complex  weather  and  the  inclusion  of  significant  terrain  features).  To  that 
end,  the  methodologies  and  reporting  techniques  described  here  represent  a  “template” 
for  future  studies. 

For  this  study,  we  adopt  Department  of  Defense  definitions  for  verification, 
validation,  and  accreditation  as  shown  below  [Ref.  1-6]: 

.  Verification  -  The  process  of  determining  the  degree  to  which  a  model  or 
simulation  implementation  accurately  represents  the  developer’s  conceptual 
description  and  specification.  Verification  also  evaluates  the  extent  to  which 
the  model  or  simulation  has  been  developed  using  sound  and  established 
software  engineering  techniques.5  [DOD  Directive  5000.59] 

.  Validation  -  The  process  of  determining  the  degree  to  which  a  model  or 
simulation  is  an  accurate  representation  of  the  real  world  from  the  perspective 
of  the  intended  uses  of  the  model  or  simulation.  [DOD  directive  5000.59] 


3  An  initial  MMO  comparison  has  recently  been  completed  [Ref.  1-3]. 

4  A  recently  completed  MMO-like  study,  which  included  HP  AC,  demonstrated  the  potential  for  these 
types  of  studies  to  identify,  clarify,  and  communicate  differences,  including  operational  differences, 
between  models  [Ref.  1-5]. 

5  This  effort  does  not  examine  software  engineering  technique  issues. 
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.  Accreditation  -  The  official  certification  that  a  model  or  simulation  is 
acceptable  for  use  for  a  specific  purpose.  [DOD  Directives  5000.59  and 
5000.59-P] 

1.  Brief  HPAC  Description 

HPAC  is  composed  of  a  suite  of  software  modules  that  can  generate  source  terms 
for  hazardous  releases,  retrieve  and  prepare  meteorological  information  for  use  in  a 
prediction,  model  the  transport  and  dispersion  (T&D)  of  the  hazardous  release  over  time, 
and  plot  and  report  the  results  of  these  calculations.  HPAC  has  been  applied  to  various 
national  defense  problems  including  military  studies  and  operational  planning. 

For  hazardous  material  T&D,  HPAC  uses  the  Second-Order  Closure  Integrated 
Puff  (SCIPUFF)  model  and  an  associated  mean  wind  field  model.  SCIPUFF,  which  is  a 
Lagrangian  model  for  atmospheric  dispersion  that  uses  the  Gaussian  puff  numerical 
method  -  an  arbitrary  time-dependent  concentration  field  is  represented  by  three- 
dimensional  Gaussian  distributions  -  bases  its  turbulent  diffusion  parameterization  on 
second-order  closure  theories.  This  methodology  provides  a  link  between  measurable 
velocity  statistics  and  the  predicted  dispersion  rates.  This  “second-order”  feature  implies 
that  concentration  variance  can  also  be  computed,  and  this  uncertainty  estimate  can  be 
used  as  the  basis  for  a  probabilistic  description  of  the  dispersion  prediction.6 

This  study  examines  the  most  recently  released  HPAC  software  version,  3.2  [Ref. 

1-8]. 


2.  Brief  NARAC  Description 

The  ADAPT/LODI7  modeling  system  is  used  for  both  real-time  operational 
applications  and  detailed  assessments  of  events  involving  atmospheric  releases  of 
hazardous  material  at  the  Department  of  Energy’s  (DOE)  National  Atmospheric  Release 
Advisory  Center  (NARAC)  at  Lawrence  Livermore  National  Laboratory  (LLNL).  The 
ADAPT  meteorological  data  assimilation  model  constructs  fields  of  such  variables  as  the 
mean  winds,  pressure,  precipitation,  temperature,  and  turbulence,  using  a  variety  of 
interpolation  methods  and  atmospheric  parameterizations  [Ref.  1-9].  Non-divergent  wind 


6  See  Reference  1-7  for  several  reports  that  provide  details  of  HPAC  design,  functionality,  capabilities, 
and  V&V. 

7  ADAPT  =  Atmospheric  Data  Assimilation  and  Parameterization  Techniques.  LODI  =  Lagrangian 
Operational  Dispersion  Integrator. 
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fields  are  produced  by  an  adjustment  procedure  based  on  the  variational  principle  and  a 
finite-element  discretization. 

The  LODI  dispersion  model  solves  the  3-D  advection-diffusion  equation  using  a 
Lagrangian  stochastic,  Monte  Carlo  method  [Ref.  1-10].  LODI  includes  methods  for 
simulating  the  processes  of  mean  wind  advection,  turbulent  diffusion,  radioactive  decay 
and  production,  first-order  chemical  reactions,  wet  deposition,  gravitational  settling,  dry 
deposition,  and  buoyant/momentum  plume  rise.  The  models  are  coupled  to  NARAC 
databases  providing  topography,  geographical  data,  chemical-biological-nuclear  agent 
properties  and  health  effects,  real-time  meteorological  observational  data,  and  global  and 
mesoscale  forecast  model  predictions. 

B.  DESCRIPTION  OF  THE  PRAIRIE  GRASS  FIELD  TRIAL  EXPERIMENT 

Prairie  Grass  field  trials  were  conducted  during  the  summer  of  1956  in  north 
central  Nebraska  near  the  town  of  O’Neil.8  The  primary  objective  of  Project  Prairie 
Grass  was  to  determine  the  rate  of  diffusion  of  a  neutrally-buoyant  tracer  gas  as  a 
function  of  meteorological  conditions.  These  experiments  involved  continuous 
ten-minute  releases  of  sulfur  dioxide  (SO2)  from  a  near-surface  point  source.  Downwind 
SO2  concentrations  were  sampled  along  five  concentric,  semi-circular  arcs  located  50, 
100,  200,  400,  and  800  meters  away  from  the  gas  source. 

The  sampling  network  utilized  midget  impingers  mounted  at  a  height  of  1.5 
meters  along  five  arcs.  In  addition,  limited  vertical  sampling  was  carried  out  along  the 
100-meter  arc  by  means  of  impingers  mounted  at  nine  levels  on  six  lightweight  towers. 
Electrically  operated  vacuum  units  suitably  positioned  within  the  sampling  network 
provided  aspiration  for  the  impingers.  During  the  diffusion  experiments,  air  was  drawn 
into  the  impingers  through  short  sections  of  capillary  tubing  and  bubbled  through  a  dilute 
hydrogen-peroxide  solution.  Sulfur  dioxide  present  in  the  air  samples  combined  with  the 
hydrogen  peroxide  to  form  sulfuric  acid.  Ten-minute  averaged  SO2  concentrations  were 
determined  from  laboratory  measurements  of  the  electrical  conductivity  of  the  aspired 
solutions.  The  samplers  were  arranged  at  2-degree  intervals  along  the  50,  100,  200,  and 
400-meter  arcs  (91  samplers  per  arc),  and  at  1-degree  intervals  for  the  800-meter  arc  (i.e., 
181  samplers  along  the  800-meter  arc). 


See  References  1-4  and  1-11. 
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The  meteorological  conditions  (wind  speed  and  direction)  were  measured  at  a 
number  of  weather  stations.  Figure  1-1  shows  the  setup  and  typical  concentrations 
recorded  at  the  Prairie  Grass  field  trials.  Surface  weather  observations  at  a  2-meter 
height  were  recorded  at  the  “Source,”  “North450,”  “MIT  3L,”  and  “MIT  AREA”  weather 
stations.  The  “A&M  TWR”  surface  tower  provided  wind  speed  and  direction  at  seven 
levels  (0.3,  0.5,  1.0,  2.0,  4.0,  8.0,  and  16.0  meters)  above  the  surface.  In  addition,  the 
“Rawinsond”  radiosonde  was  used  to  provide  meteorological  conditions  at  upper  levels. 


Figure  1-1.  Example  Dosages  (in  mg-sec/m3)  and  Field  Trial  Setup9 

Approximately  70  experiments  were  carried  out  in  a  wide  variety  of  weather 
conditions.  Some  of  the  trials  (30,  31,  63,  and  64)  did  not  include  correction  factors  for 
the  compensation  of  the  evaporative  loss  of  impinger  solution  during  aspiration.9 10  Other 


9  The  dosages  shown  here  were  obtained  from  Prairie  Grass  field  trial  61.  The  color  intensity 
corresponds  to  the  logarithm  of  dosage  at  the  samplers.  Dosages  are  calculated  by  multiplying  ten- 
minute  averaged  SCE  concentrations,  as  reported  in  the  Prairie  Grass  field  trials,  by  600  seconds.  The 
source  release  point  is  at  (535.2  km,  4704.427  km).  The  sampling  network  is  oriented  on  a  West-East 
grid,  with  the  middle  sampler  pointing  toward  the  north.  Sampler  1  corresponds  to  the  west-most 
sampler.  Red  symbols  denote  locations  of  the  meteorological  stations. 

10  See  Reference  1-4,  pages  200-201. 
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Prairie  Grass  trials  have  significant  portions  of  meteorological  data  missing.  For  these 
reasons,  19  trials  were  removed  from  this  comparison.  Appendix  B  gives  further  details 
for  the  elimination  of  the  Prairie  Grass  trials.  Table  1-1  lists  trials  that  were  used  in  this 
HPAC/NARAC  comparison. 

Table  1-1.  List  of  Prairie  Grass  Trials  That  Were  Used  in  HPAC/NARAC  Comparison 

(5,  7,  8,  9,  10,  11,  12,  13,  14,  15,  16,  17,  18,  19,  20,  21, 22,  23,  24,  25,  26,  27,  28,  32,  33,  34,  35, 
36,  37,  38,  39,  40,  41 , 42,  43,  44,  45,  46,  48,  49,  50,  51 , 54,  55,  56,  57,  58,  59,  60,  61 ,  62} 


On  some  trials,  critical  information  associated  with  a  specific  sampler  arc  was 
missing.  The  50-meter  arc  of  trial  62  did  not  include  a  correction  factor  for  the 
compensation  of  the  evaporative  loss.  The  200-meter  arc  of  trial  50  and  the  100-meter 
arc  of  trial  57  had  vacuum  lines  disconnected.11  We  eliminated  these  three  arcs  from  our 
HPAC/NARAC  comparisons. 

Computer-ready  data  files  of  the  Prairie  Grass  field  trials  were  provided  by 
Lawrence  Livermore  National  Laboratory  [Ref.  1-12].  Appendix  K  provides  a  list  of 
data  anomalies  that  we  discovered  during  our  analysis  of  the  Prairie  Grass  field  trial  data 
and  the  “corrections”  that  we  applied. 

C.  NEAR-TERM  PLANNED  COMPARISONS 

Follow-on  analyses,  planned  for  the  near-term,  will  describe  the  results  of 
comparisons  of  these  models  to  field  trial  data  collected  during  the  Dugway  Proving 
Ground  Phase  I  experiment  [Ref.  1-13].  Further  analysis  that  considers  the  usage  of 
hazard  prediction  models  with  more  limited  weather  inputs  and  probabilistic  outputs  is 
planned  and  will  include  the  application  of  the  proposed  user-oriented  MOEs.  We  expect 
that  future  examinations  will  include  field  trial  data  collected  over  longer  ranges  and 
complex  terrain. 


1 1  See  Reference  1-4,  pages  79-80. 
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CHAPTER  2 

METHODOLOGIES 


2.  METHODOLOGIES 


This  chapter  provides  a  description  of  the  methodologies  used  to  compare  HP  AC 
and  NARAC  predictions  to  the  Prairie  Grass  field  trial  observations.  In  particular,  two 
new  user-oriented  MOEs  are  defined  and  discussed.  Standard  statistical  measures,  which 
have  been  used  to  compare  hazard  prediction  model  output  with  field  trial  experiments  in 
the  past,  are  also  described.  In  addition,  the  conditions  associated  with  the  Prairie  Grass 
field  trial  observations  that  were  used  to  compare  the  models  are  presented  in  this 
chapter.1 

A.  BACKGROUND 

In  general,  model  validation  efforts  include  specific  measures  of  effectiveness  that 
are  needed  to  define  a  metric  by  which  field  trial  observations  and  predictions  can  be 
compared.  It  would  also  be  helpful  if  model  accreditation  were  to  include  measures  of 
effectiveness  that  relate  “operational”  use  of  the  model  to  field  trial  experiments.  Such 
MOEs  would  give  a  certain  degree  of  confidence  to  users  with  respect  to  how  closely  the 
model  approximates  the  real  world  in  their  particular  situation. 

In  this  paper,  we  introduce  a  family  of  novel  MOEs  that  could  aid  both  validation 
and  accreditation  efforts.  For  the  most  part,  the  analyses  and  discussion  described  in  this 
paper  are  associated  with  validation.  However,  the  application  of  these  MOEs  begins  to 
address  issues  associated  with  user  accreditation. 

All  methods  of  evaluating  model  performance  have  both  strengths  and 
limitations.2  Any  single  statistical  or  graphical  measure  of  model  performance  involves 
explicit  and  implicit  judgments  as  to  the  most  important  aspect(s)  of  model  performance 
(e.g.,  the  peak  concentration,  the  integrated  dose,  the  plume  arrival  time,  the  location  of 
affected  populations,  or  the  correspondence  between  calculated  and  measured  values  at 
all  locations).  Statistical  measures  can  also  be  sensitive  to  details  of  the  comparison 
calculations,  including  the  selection  and/or  elimination  of  data  (e.g.,  the  removal  of  data 


1  Appendix  B  contains  additional  details  that  describe  the  inputs  for  the  HP  AC  and  NARAC  predictions. 

2  For  example,  see  Ref.  2- 1 . 


2-1 


based  on  a  specified  threshold  value),  the  data  transformations  applied  (e.g.,  the  use  of 
logarithmic  data  transformations,  time  or  space  averaging),  and  the  criteria  used  to  pair 
data  (e.g.,  the  pairing  of  observed  and  predicted  values  at  the  same  point  in  space  and 
time,  the  elimination  of  directional  effects  by  the  use  of  derived  quantities  such  as  the 
integrated  dose  or  the  peak  value).  The  weakness  of  any  single  measure  can  be 
exacerbated  by  the  limited  resolution  of  the  plume  provided  by  sparsely  distributed 
observations  (in  time  or  space)  and  the  strong  correlation  of  values  across  the  entire 
plume,  which  limits  the  validity  of  assumptions  about  data  independence.  Therefore, 
overall  model  performance  analysis  should  involve  many  measures  and  is  an  iterative 
process  in  which  selected  statistical  and  graphical  analyses  are  used  to  formulate  specific 
questions,  identify  differences  in  model  results,  and  test  trends. 

Graphical  comparisons  (e.g.,  overlays  of  measurements  on  two-dimensional 
predicted  plume  plots,  plots  of  predicted  and  observed  dosages  along  sampler  arcs,  time- 
histories  of  predicted  and  observed  dosages  at  a  given  sampler  location)  may  be  the  most 
generally  useful  and  easily  interpretable  methods  of  evaluation.  These  types  of  plots 
allow  the  user  to  detect  patterns  that  are  not  easily  reduced  to  statistical  or  numerical 
comparisons  and  to  both  formulate  and  verify  conclusions  about  plume  direction,  plume 
spread,  and  peak  concentrations.  However,  comparisons  based  on  graphical  patterns  can 
be  sensitive  to  the  choice  of  contour  values  or  dosage  ranges,  particularly  when 
comparing  data  that  range  over  several  orders  of  magnitude.  In  addition,  graphical 
evaluations,  by  their  very  nature,  tend  to  be  subjective  and  less  quantitative. 

Statistical  measures  potentially  allow  for  more  objective  quantitative  evaluation 
of  model  performance.  They  also  allow  overall  performance  for  a  large  number  of  data 
points  to  be  assessed,  along  with  the  statistical  significance  of  differences.  However, 
standard  statistical  measures  (e.g.,  fractional  bias,  geometric  bias,  normalized  mean 
square  error)  suffer  from  various  shortcomings,  including  a  potential  over- sensitivity  to 
either  low  or  high  data  values.  All  such  measures  can  be  heavily  influenced  by  the 
criteria  used  to  eliminate  and/or  combine  data  (e.g.,  choice  of  thresholds,  methods  for 
handling  zero  values).  Typically,  the  most  robust  approach  is  the  use  of  statistical 
measures  in  conjunction  with  insight  gained  from  examination  of  graphical  plots  of 
observed  and  predicted  data. 

In  general,  a  single  method  of  evaluation  or  statistical  measure  should  not  be  used 
in  isolation.  Application  of  any  measure  requires  a  clear  understanding  and 
documentation  as  to  the  purpose,  meaning,  and  limitations  of  the  comparison  criteria,  as 
well  as  the  methodology  by  which  it  was  calculated.  Multiple  measures  need  to  be 
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available  to  meet  different  user  needs  and  to  conduct  different  types  of  analysis.  In 
general,  the  most  robust  and  perhaps  the  only  valid  conclusions  concerning  model 
differences  are  those  supported  by  a  number  of  complementary  measures. 

B.  COMPARATIVE  MEASURES 

Several  recent  comparisons  of  T&D  model  predictions  to  field  data  and  other 
models  have  demonstrated  a  variety  of  measures  and  issues  [Refs.  2-2  through  2-10].  In 
particular,  it  has  been  pointed  out  [Ref.  2-10]  that  the  relative  advantages  of  different 
statistical  measures  of  performances  with  respect  to  accurately  communicating  model 
behavior  is  dependent,  at  least  in  part,  on  the  distribution  of  the  variables  of  interest.  In 
this  study,  we  compute  several  statistical  measures  of  performance  and  user-oriented 
measures  of  effectiveness  (MOE)  and  base  our  conclusions  on  a  review  of  all  of  these 
measures. 

The  rest  of  this  section  provides  a  detailed  description  of  the  two  MOEs  we 
examined  and  a  brief  description  of  the  statistical  measures  we  computed. 

1.  Measures  of  Effectiveness 

Interest  in  developing  application-specific  MOEs  is  ultimately  related  to  the 
desire  to  improve  the  potential  for  validation  and  user  accreditation.3  Typical  measures, 
or  figures  of  merit  (FOM),  use  ratios  or  differences  between  predicted  and  observed 
quantities  (e.g.,  dosages  or  plume  centerline).  In  general,  these  FOMs  do  not  necessarily 
reflect  specific  user  application-based  concerns. 

Perhaps  the  key  objective  for  an  interpretive  user-oriented  MOE,  is  its  ability  to 
communicate  simply  (and  appropriately)  the  relative  performance  of  the  model  in  a  given 
application.  It  is  important  that  other  measures,  for  instance,  the  statistical  measures 
described  later  in  this  paper,  also  be  examined  for  any  set  of  data  that  is  to  be  described. 
In  order  for  such  an  MOE  to  have  widespread  credibility,  the  conclusions  communicated 
to  the  user  via  the  MOE  should  be  consistent  with  the  conclusions  one  would  obtain  from 
a  detailed  review  of  other  statistical  measures  and  graphical  representations.  Where  there 
are  inconsistencies,  the  differences  should  be  highlighted  and  understood. 

In  the  next  six  subsections,  we  define  and  explore  two  MOEs  that  have  the 
potential  to  address  user-specific  requirements  directly. 


3  See  Appendix  N,  “Task  Order  Extract”  and  Reference  2-11. 
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a.  User-Oriented  Measure  of  Effectiveness:  MOE  1 


A  fundamental  feature  of  any  comparison  of  model  output  to  observations  is  the 
over-  and  under-prediction  regions.  We  can  define  false  negative  where  a  hazard  is 
observed  but  not  predicted  and  false  positive  where  a  hazard  is  predicted  but  not 
observed. 

Figure  2-1  shows  the  observed  area  at  a  given  dosage  level  and  the  predicted  area 
at  the  same  dosage  level  for  some  hypothetical  situation.  Numerical  values  in  three- 
dimensional  space  characterize  this  conceptual  view.  These  values  are  associated  with 
estimates  of  the  false  negative  region  (Arm),  the  false  positive  region  (AFP),  and  the 
overlap  region  (A0v)-  Figure  2-1  illustrates,  conceptually,  Aov,  Arm,  and  App. 

The  first  measure  that  we  describe  considers  the  ratio  of  the  area  of  overlap  (Aov) 
to  the  total  area  (AT).  Aov  is  the  area  in  which  both  the  prediction  and  the  observation 
agree  in  some  manner;  for  example,  both  the  predictions  and  the  observations  show 
dosages  above  a  certain  level.  Graphically,  A0v  might  correspond  to  the  intersection  of 
the  two  areas  for  which  a  certain  dosage  is  exceeded. 


Figure  2-1.  Illustration  of  AFN,  AFP,  and  A0v 

At  includes  the  overlap  area  (Aov),  the  area  in  which  a  certain  dosage  level  was 
predicted  but  not  observed  (“false  positive”),  and  the  area  in  which  a  certain  dosage  was 
observed  but  not  predicted  (“false  negative”).  Equation  2-1  describes  this  MOE.  This 
measure  for  evaluating  transport  and  dispersion  models  has  been  previously  described 
[Refs.  2-12  and  2-13]. 
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where  Afn  is  the  false  negative  area  and  AFP  is  the  false  positive  area. 


Typical  operational  users  of  transport  and  dispersion  models  might  consider  false 
positives  and  false  negatives  quite  differently.  For  many  applications,  false  positives 
would  be  much  more  acceptable  to  the  user  than  false  negatives  (which  could  result  in 
decisions  that  directly  lead  to  death  or  injury).  It  has  been  suggested  that  equation  2-1  be 
modified  with  “tunable”  coefficients  that  allow  a  user  to  weight  the  relative  importance 
of  false  negatives  and  false  positives  [Ref.  2-14].  Equation  2-2  illustrates  this 
modification,  with  Cfn  defined  as  the  false  negative  coefficient  and  Cpp  defined  as  the 
false  positive  coefficient. 


lov 


(A 


OV  ~  ^FN-^FN  ~  ^FP^FP ) 


(2-2) 


where  Cfn,  Cfp  >  0. 

It  may  be  true  that,  for  some  comparisons  of  predictions  and  observations,  the 
weightings  for  false  negatives  and  false  positives  are  considered  irrelevant  or  set  equal 
(Cfn  =  Cfp).4 

Figure  2-2  describes  the  behavior  of  MOE  1  (Crn  =  CFP  =  1.0)  as  a  function  of 
AFn  and  AFP.  For  the  computation  of  the  values  shown  in  Figure  2-2,  AT  was  normalized 
to  1.0.  That  is,  AT  =  Aov  +  Afn  +  App  =  1.0.  Thus,  the  values  of  AFn  and  AFP  shown  on 
the  x-  and  y-axis  correspond  to  the  fraction  of  the  total  area  that  is  represented  by  false 
negative  and  false  positive,  respectively. 

It  can  be  seen  that  MOE  1  with  Cfn  =  CFP  =  1.0,  varies  linearly  from  0.0  to  1.0  as 
AFn  and  AFP  decrease.  A  perfect  one-to-one  correspondence  between  a  model’s 
predictions  and  observations  would  result  in  an  MOE  1  value  of  1.0. 


4  As  developed  here,  the  implicit  coefficient  associated  with  A0v  is  1.0.  Therefore,  the  notion  of  equal 
weights  for  A™  and  AFP  (i.e.,  Cfn  =  CFp)  is  insufficient  for  the  complete  specification  of  this  MOE. 
That  is,  the  precise  MOE  values  will  depend  on  the  values  chosen  for  CFn  and  CFP  and  not  just  their 
ratio. 
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Figure  2-2.  MOE  1  as  a  Function  of  AFN  and  AFP  (CFN  =  CFP  =1.0) 


Figure  2-3  illustrates  the  shape  of  this  MOE  when  the  coefficients  are  varied.  The 
somewhat  extreme  weighting  (i.e.,  100  to  1)  of  false  negative  relative  to  false  positive 
and  overlap  area  shown  in  the  bottom  plot  of  Figure  2-3  illustrates  the  potential  for 
significant  nonlinear  behavior  of  this  measure.  We  expect  that  measures  that  perform 
linearly  (or  nearly  linearly)  over  a  large  range  will  have  greater  utility  than  those  that  do 
not,  at  least  with  respect  to  the  communication  of  model  performance.  It  is  also  true  that 
highly  nonlinear  measures  could  amplify  relatively  small  changes  in  a  data  set,  perhaps 
due  to  a  minor  measurement/transcription  error.  Therefore,  such  a  nonlinear  measure 
might  not  be  considered  robust. 
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Figure  2-3.  MOE  1  as  a  Function  of  CFN  and  CFP 
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b.  A  Two-Dimensional  MOE:  MOE  2 


The  second  MOE  that  we  consider  is  two-dimensional.  That  is,  it  is  represented 
by  a  pair  of  numbers  (x,y).  First,  we  consider  the  ratios  of  the  overlap  area  (Aov)  to  the 
observed  and  predicted  areas.  Equation  2-3  describes  these  ratios  as  a  point  in 
two-dimensional  space. 


A  A 

^ov  ^ov 


A  A 

^ OB  ^ PR 


(2-3) 


where  Aob  is  the  area  associated  with  the  observations  and  APR  is  the  area  associated  with 
the  prediction.  Inspection  of  Figure  2-1  suggests  the  relationships  Aob  =  Aov  +  Arn  and 
Apr  =  Aov  +  App.  Substituting  these  equalities  into  the  denominator  of  equation  2-3, 
followed  by  algebraic  rearrangement  leads  to  Equation  2-4  -  our  preferred  definition  of 
MOE  2. 
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A  model  prediction  that  is  in  perfect  agreement  with  an  observation,  that  is,  all 
overlap  area  and  no  false  positive  or  false  negative,  will  have  an  MOE  2  value  of  (1,1). 
A  model  prediction  that  entirely  misses  the  observed  cloud,  perhaps  because  of  an  errant 
wind  direction,  would  have  no  overlap  and  an  MOE  2  value  of  (0,0). 


c.  Interpretation  of  MOE  2 

MOE  2  separates,  orthogonally,  the  two  fundamental  features  of  any  transport  and 
dispersion  predictive  model  -  “false  negatives”  and  “false  positives.”  The  user  or 
specific  application  determines  the  acceptability  of  a  given  false  positive  or  false  negative 
fraction.  In  practice,  one  expects  that,  for  a  given  model,  various  underlying  assumptions 
about  input  uncertainties  (for  example)  can  affect  the  relative  ratio  of  A  i  n  and  App. 
Without  some  fundamental  improvement  to  the  model,  one  expects  that  decreases  in  App 
will  lead  to  increases  in  Arm.  Similarly,  decreases  in  Arn  will  be  achieved  only  at  the 
expense  of  increases  in  APP  (without  some  fundamental  model  improvement,  for  example 
better  weather  data).5 


5  More  accurate  weather  data,  for  example,  a  more  accurate  wind  direction,  might  improve  both  the  false 
negative  and  false  positive  behavior  of  a  prediction.  That  is,  the  2D  MOE  value  would  move  along  the 
diagonal  up  and  to  the  right  (Figure  2-4). 
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Figure  2-4  illustrates  the  potential  of  MOE  2  for  assessing  application-  or  user- 
specific  worth  for  a  T&D  model.  The  axes  of  the  figure  are  as  shown  in  equation  2-4. 
Along  the  x-axis,  the  fraction  of  the  false  negative  area  (Afn)  decreases  from  left  to  right. 
That  is,  model  performance  improves  as  this  value  gets  larger.  Similarly,  as  the  values 
move  up  the  y-axis,  a  decrease  in  App  is  indicated. 
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Figure  2-4.  MOE  2  as  a  Function  of  AFN  and  AFP 

We  imagine  a  given  user  and  application  being  associated  with  some  risk 
tolerances,  ri  and  T2,  and  therefore  accept  from  a  given  prediction  a  certain  false  positive 
or  false  negative  fraction.  In  essence,  these  risk  tolerances  define,  in  the  MOE  space,  an 
acceptable  or  desirable  region  for  model  performance. 

We  envision  that  users  may  have  very  different  requirements.  For  example,  one 
application  may  concern  estimating  the  hazardous  area  to  a  population  and  deciding  how 
many  people  to  evacuate.  Clearly,  large  amounts  of  false  negative  area  would  not  be 
acceptable  -  many  people  might  be  unnecessarily  exposed.  Alternatively,  too  large  an 
area  of  false  positive  would  be  inefficient  and  perhaps  dangerous  if  it  implied  the 
evacuation  of  a  large  population.  A  similar  argument  can  be  made  for  the  use  of  a  model 
to  predict  the  release  of  hazardous  material  from  an  attack  on  an  enemy  CB  facility. 
Again,  a  low  false  negative  fraction  would  be  required,  but  a  large  false  positive  fraction 
could  greatly  limit  the  target  set,  and  adversely  impact  the  wartime  strategy. 

Finally,  one  can  imagine,  for  instance,  a  CB  ballistic  missile  attack  in  the  vicinity 
of  a  U.S.  military  unit.  In  such  a  case,  an  immediate  goal  of  a  friendly  chemical 
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reconnaissance  unit  would  be  to  locate  the  hazardous  area,  perhaps  with  only  limited 
information,  and  ensure  that  future  troop  movements  avoid  this  area.  For  this  mission 
then,  this  reconnaissance  unit  would  want  to  encounter  the  chemical  agent  quickly  in 
order  to  start  mapping  the  hazardous  location.  Unlike  the  other  scenarios  discussed 
above,  a  T&D  predictive  system  with  a  relatively  high  false  negative  fraction,  but  with  a 
very  low  false  positive  fraction,  might  be  considered  quite  satisfactory  for  this 
application. 

An  additional  feature  associated  with  this  2D  MOE  is  highlighted  by  the  dashed 
purple  line  that  lies  along  the  diagonal  in  Figure  2-4.  Setting  x\  =  X2,  we  can  derive  from 
equation  2-4  the  equation  shown  below. 
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Therefore,  MOE  2  values  lying  on  the  diagonal  line  represent  cases  where  the  total 
predicted  and  observed  plume  areas  are  of  equal  size.  For  example,  an  estimate  of  (1,1), 
at  the  top  of  diagonal,  corresponds  to  a  predicted  and  observed  plume  of  the  same  size 
that  exactly  coincide  -  that  is,  perfect  overlap.  A  2D  MOE  estimate  of  (0,0),  at  the  lower 
end  of  the  diagonal,  implies  plumes  that  miss  each  other  completely  -  that  is,  no  overlap 
(perhaps  due  to  incorrect  wind  direction  information).6  As  MOE  values  traverse  this 
diagonal,  from  (0,0)  to  (1,1),  the  suggestion  is,  at  least  for  equally  shaped  plumes,  that  the 
wind  field  model  components  (e.g.,  direction  and  speed)  have  improved  (i.e.,  the  amount 
of  overlap  is  increasing). 

MOE  2  values  along  the  axis  (l,y)  imply  no  false  negative  fraction.  That  is,  the 
prediction  totally  includes  the  observation  (e.g.,  a  probabilistic  output  that  considered  an 
ensemble  of  initial  conditions  might  be  represented  on  this  axis).  Along  the  axis  defined 
by  (x,l),  the  suggestion  is  that  the  predicted  area  fits  totally  within  the  observed  area, 
with  the  false  negative  fraction  described  by  1-x. 

In  the  end,  the  credibility  and  usefulness  of  MOEs  designed  to  incorporate 
user/application-specific  needs  will  require  review  by  the  user  -  as  part  of  the 


6  Mathematically,  at  exactly  (0,0)  nothing  can  be  inferred  about  the  sizes  of  the  plumes.  However,  at 
any  point  arbitrarily  close  to  (0,0)  and  along  the  diagonal,  it  will  be  true  that  the  predicted  and 
observed  area  sizes  will  be  equal  with  almost  no  overlap. 
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accreditation  process.  We  believe  that  measures  that  clearly  present  the  “tradeoffs” 
associated  with  false  negatives  and  false  positives  stand  the  best  chance  to  satisfy  the 
user’s  accreditation  needs. 


Figure  2-5  suggests  an  additional  interpretation  of  the  2D  MOE.  In  this  figure, 
the  multi-pointed  gold  star  represents  the  estimate  of  MOE  2  for  some  set  of  fictional 
model  predictions  and  field  trial  observations.  The  point  estimate,  perhaps  the  vector 
mean  value  of  several  similar  trials,  would  be  found  at  the  center  of  this  star,  and  the 
overall  size  of  the  star  represents  the  uncertainty  associated  with  the  point  estimate  of 
MOE  2.  If  a  second  set  of  model  predictions  were  compared  to  “Model  A”  with  MOE  2, 
several  conclusions  might  be  anticipated.  The  second  model  estimate  might  be  found  in 
the  region  shaded  orange  (lower  left).  This  would  imply  that  Model  A  performs 
significantly  better;  both  its  false  positive  and  false  negative  fractions  are  lower. 
Alternatively,  the  second  model  might  lead  to  an  estimate  in  the  green  region  (upper 
right)  -  an  indication  that  Model  A  is  the  poorer  performer  (for  this  set  of  field  trial 
observations).  Finally,  the  new  model  predictions  might  lead  to  an  MOE  2  value  that  is 
located  within  the  gray  region.  The  implication  here  is  that  a  user  would  have  to  make  a 
determination  as  to  the  tradeoff  between  false  positive  and  false  negative  before  deciding 
which  model  was  most  appropriate  for  his/her  specific  application. 
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Figure  2-5.  2D  MOE:  Interpretation  of  Model  Comparisons 
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d.  Relationship  Between  MOE  1  and  MOE  2 


The  two  MOEs  described  previously,  one-  and  two-dimensional,  are  related. 
Equation  2-6  relates  the  x-  and  y-axes  of  the  2D  MOE  to  the  one-dimensional  MOE. 


MOE  1  = 


_ ry _ 

(xy  +  CFN  y(l  -  x)  +  CFPx{  1  -  y )) 


(2-6)7 


Figure  2-6  shows  MOE  contours  (i.e.,  isolines  for  MOE  1)  for  Cfn  =  Cfp  =  1. 
Similarly,  Figure  2-7  illustrates  the  case  where  Afn  is  weighted  by  a  factor  of  10  relative 
to  App  and  5  relative  to  Aov  (he.,  Cfn  =  5  and  Cfp  =  0.5). 
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Figure  2-6.  Relationship  Between  MOE  1  and  MOE  2:  CFn  =  1,  CFP  =  1 
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Figure  2-7.  Relationship  Between  MOE  1  and  MOE  2:  Cfn  =  5,  Cfp  =  0.5 
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These  definitions  of  AFN  and  AFP  are  then  substituted  into  equation  2-2. 
rearrangement,  equation  2-6  is  obtained. 
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The  above  two  figures  show  that  estimates  of  the  2D  MOE  can  be  directly 
converted  to  estimates  of  MOE  1.  For  example,  a  2D  MOE  point  estimate  of  (0.5, 0.5) 
implies,  according  to  Figure  2-6  (and  equation  2-5),  an  MOE  1  value  of  0.33  for  Cfn  = 
Cfp  =  1.0.  Similarly,  inspection  of  Figure  2-7  shows  that  for  Cfn  =  5  and  Cfp  =  0.5,  a  2D 
MOE  point  estimate  of  (0.5, 0.5)  suggests  an  MOE  1  value  of  about  0.15. 

e.  Two  Estimates  of  “Area”  Size 

A  1998  comparison  of  long-range  dispersion  model  predictions  included  an  MOE 
similar  to  the  one-dimensional  measure  (MOE  1)  that  was  described  previously.  In  that 
case  [Ref.  2-13],  the  measure  was  referred  to  as  the  figure-of-merit  in  space  and  did  not 
include  weighting  coefficients  (Cfn  and  Cfp).  Long-range  observations  (across  Europe) 
from  the  European  Tracer  Experiment  (ETEX)  were  used  to  develop  contours  (isolines) 
at  given  concentrations  using  logarithmic  interpolation  of  the  ground-level  sampled 
values.  This  measure  was  deemed  an  effective  index  for  describing  model  performance 
at  a  fixed  time,  although  it  was  suggested  that  this  figure-of-merit  in  space  was  sensitive 
to  the  type  of  interpolation  scheme  used. 

Often,  field  trial  observations  are  represented  by  dosages  at  specific  sampler 
locations,  perhaps  along  an  arc  or  line.  Therefore,  a  direct  estimate  of  the  area  that 
contains  a  given  concentration  or  dosage  is  not  available.  When  using  the  above  MOEs 
for  comparisons  to  observations,  an  approach  to  estimating  the  “area”  size  and  location 
must  be  developed. 

Below  we  describe  two  different  methods  that  can  be  used  to  develop  “area” 
estimates  from  Prairie  Grass  observations. 

1)  Area  Estimate  1  (AE1) 

AE1  is  illustrated  in  Figure  2-8  for  the  800-meter  arc  samplers  of  Prairie  Grass 
field  trial  8.  First,  we  define  a  threshold  dosage  of  interest.  In  the  figure,  a  threshold  of 
60  mg-sec/m  is  used.  One  then  can  develop  an  area  estimate,  referred  to  here  as  AE1, 
say  for  the  false  positive,  by  computing  the  distance  covered  by  the  samplers  in  which 
there  is  a  prediction  but  no  observation  above  threshold.  Similarly,  distances  for  the 
overlap  region  and  false  negative  region  can  be  computed.  These  distances  are  naturally 
related  to  actual  areas.  That  is,  the  AE1  estimate  represents  a  natural  analog  to  a  hazard 
area  at  a  given  critical  threshold. 
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Prairie  Grass  Trial  8,  ARC800 


Figure  2-8.  Illustration  of  AE1 


2)  Area  Estimate  2  (AE2) 

AE2,  illustrated  in  Figure  2-9,  can  be  computed  by  integrating  the  areas  under  the 
associated  curves.  For  example,  for  App  (as  shown  in  Figure  2-9),  one  first  considers  all 
of  the  samplers  at  which  the  prediction  is  of  greater  value  than  the  observation.  Next, 
one  simply  sums  the  differences  between  the  predicted  and  observed  dosages  at  those 
samplers.  Based  on  the  samplers  that  contained  observed  values  that  were  larger  than 
the  predicted  values,  one  can  similarly  compute  Afn.  Aov  is  calculated  by  considering 
all  samplers  and  summing  the  dosages  associated  with  the  minimum  predicted  or 
observed  value.  Aov,  App,  and  Afn  are  computed  as  shown  in  Figure  2-9  and  are 
somewhat  related  to  crosswind  integrated  (CWI)  dosages.  However,  AE2  involves  point- 
to-point  comparisons  and  includes  directional  effects,  unlike  CWI  dosages. 

It  should  be  noted  that  the  terms  “false  negative”  and  “false  positive,”  as  defined 
in  Figure  2-1  and  used  in  AE1,  are  not  strictly  applicable  to  AE2.  A  more  accurate 
description  is  that  AE2  measures  the  degree  of  under-prediction  and  over-prediction  of 
dosages  integrated  along  the  sampler  arc.  Unlike  an  MOE  using  AE1,  a  perfect  MOE 
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score  based  on  AE2  indicates  “exact”  agreement  between  measured  and  predicted 
dosages  at  every  sampler  location.8 
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Prairie  Grass  Trial  8,  ARC800 


Figure  2-9.  Illustration  of  AE2 


f.  Application  of  Probit  Curves  as  a  “Lethality/Effects  Filter” 

Next,  we  interpret  observations  and  predictions  by  considering  the  lethality  or 
effects  of  a  presumed  agent.  First,  the  lethality/effects  of  the  agent  being  studied  and  the 
dosages  at  a  given  sampler  are  examined.  For  example,  standard  probit  curves  might  be 
used  to  assess  the  lethality/effects  of  a  given  exposure.9  Next,  the  dosages  are  converted 


8  Theoretically,  the  computation  of  AE2 -based  values  does  not  require  the  use  of  a  threshold.  However, 
field  trial  data  will  always  have  uncertainties  associated  with  the  smallest  observations  -  that  is, 
experimental  resolution.  Therefore,  AE2 -based  computations  that  use  field  trial  data  require  a  defined 
threshold.  Thus,  for  a  perfect  MOE  2  AE2  value  of  (1,1),  exact  agreement  at  samplers  above  some 
threshold  can  be  inferred. 

9  Lethality  and  effects  models  may  be  subject  to  poorly  known  thresholds  and  probit  slopes  (assuming  a 
probit  model).  There  may  be  uncertainties  in  the  knowledge  of  the  source  purity  and  unknown 
correction  factors  for  applying  data  to  a  general  population  versus  military  personnel.  Determining 
actual  casualty  levels  for  a  specific  scenario  would  also  need  to  include  factors  such  as  the  means  of 
exposure  (respiratory  or  percutaneous)  and  other  parameters  relating  to  that  exposure  (e.g.,  breathing 
rate  and  the  effect  of  clothing). 
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into  a  fractional  lethal/effects  exposure,  i.e.,  that  fraction  of  an  exposed  population,  at  that 
dosage  level,  that  would  be  expected  to  become  a  casualty.  For  example,  if  the  observed 
or  predicted  dosage  were  LD5o  (lethal  dosage  for  50  percent),  then  half  of  the  population 
would  be  expected  to  die. 

Figure  2-10  provides  a  nominal  illustration  of  this  calculation.  First,  the  observed 
dosage  at  sampler  “17”  is  10  LD5oS  (10  x  LD5o)  and  the  predicted  dosage  (by  HP  AC  in 
this  illustration)  is  1,000  LD50S.  These  differences  can  be  converted  into  the  fraction  of 
the  exposed  population  that  would  be  expected  to  become  a  casualty.  Assume  that  for  the 
probit  curve  associated  with  the  hypothetical  agent,  a  dosage  of  10  x  LD50  means  that  90 
percent  of  the  population  is  exposed  to  a  lethal  dosage,  while  a  dosage  of  1000  x  LD50 
implies  99.99  percent  of  the  population  accumulates  a  lethal  dosage.  The  10  LD50S 
observed  dosage  indicates  that  0.90  of  the  exposed  population  would  be  expected  to 
accumulate  a  lethal  dosage.  In  this  case,  the  model  overpredicts  the  observation  and  the 
area  of  false  positive,  AFp,  corresponds  to  an  additional  (approximately)  0.10  lethal 
exposure  population  fraction.10  Then  for  each  trial  (or  arc  or  sampler),  estimates  of  Aov, 
Afn,  and  App  can  be  obtained  in  this  way. 


Dosage 


Afp(17)  =  Additional  fraction  of  people  that 
get  lethal  exposure  after  inhaling  additional 
990  x  LDS0  (additional  9.99%  of  population) 
as  predicted  by  HPAC 


Aov(17)  =  Fraction  of  people  that 
get  lethal  exposure  after  inhaling 
10  x  LDS0  (=  ED(J())  as  OBSERVED 


Figure  2-10.  Nominal  Lethality/Effects  Filter  Illustrated 


10  In  this  study,  the  underlying  population  distribution  was  assumed  to  be  spatially  uniform  for  the 
computation  of  the  lethality/effects-filtered  MOEs. 
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In  a  sense,  the  above  process  filters  our  interpretation  of  the  above  areas  (i.e., 
Aov,  A hp,  and  Afn)  through  the  appropriate  lethality/effects  “lens.”  Hence,  we  refer  to 
MOE  results  obtained  in  this  way  as  “lethality/effects-filtered.” 


Figure  2-11  suggests  that  these  lethality/effects-filtered  results  can  be  viewed  as 
an  extension  of  AE2.  By  accounting  for  the  lethality/effects  of  a  given  hazard,  this 
measure  operationally  weights  the  over-  or  under-prediction  of  a  model  (with  respect  to  a 
set  of  observations).  Therefore,  differences  between  observations  and  predictions  that 
would  lead  to  additional  deaths  or  casualties  are  considered  more  important  than  those 
that,  perhaps,  lead  to  very  different  dosages  but  a  similar  fraction  of  lethal/effects 
exposure. 


Prairie  Grass  Trial  8,  ARC800 


Extension  of  AE2  concept 


Assume  that  the  given  substance  has 
LDS0  =  x  and  probit  slope  =  a. 

Then,  for  any  dosage  d,  let 

LE (d)  be  the  fraction  of  people  that 

get  a  lethal  exposure. 


For  any  sampler  s,  let 

Aov(s)  =  I  .E(M  i  n  ( HP  A  C(.s),observed(.v))) 

AFp(s)  =  Max(LE( HPAC(.v))-  Aov(s),0) 

Afn(s)  =  MaxILEIOBSIs))-  Aovfr),0) 

Figure  2-11.  Lethality/Effects  Filter  as  an  Extension  of  AE2 

For  this  comparison  to  Prairie  Grass  trials,  the  tracer  agent  was  sulfur  dioxide, 
SOi.  Because  this  agent  is  considered  relatively  non-toxic,  one  cannot  directly  compute 
lethality/effects  filter-based  MOEs.  Of  course,  for  some  assumed  lethality/effects  levels, 
one  can  apply  the  above  methodology  to  the  Prairie  Grass  observations  (as  we  do  in 
Chapter  3  in  order  to  demonstrate  the  utility  of  this  method).11 


11  In  Chapter  3,  the  relationship  between  lethality/effects-filtered  MOE  results  and  AE1/AE2  results  will 
be  described. 
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2.  Statistical  Measures 


In  addition  to  the  MOEs  described  above,  eight  other  statistical  measures  of 
performance  were  computed  for  the  model  comparisons  described  in  this  paper.  Two 
derived  parameters  were  considered  for  the  calculation  of  each  of  these  statistical 
measures.  First,  for  each  arc  the  maximum  dosage  was  considered  -  ArcMax.  Next,  the 
crosswind-integrated  (CWI)  dosage  was  examined. 


The  statistical  measures  include  fractional  bias  (FB),  geometric  mean  bias  (MG), 
normalized  mean  square  error  (NMSE),  geometric  mean  variance  (VG),  correlation 
coefficient  (R),  fraction  of  predictions  within  a  factor  of  2  (FAC2),  fraction  of  predictions 
within  a  factor  of  5  (FAC5),  and  fraction  of  predictions  within  a  factor  of  10  (FAC  10). 
These  measures  are  defined  below: 


FB  = 


(c,-cj 

0.5(C„+C„) 


(2-7) 


MG  =  exp  (in  Ca  -  In  Cp )  =  exp 
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FAC(x)  =  fraction  of  data  for  which  —  <  — —  <  x 
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(2-8) 

(2-9) 

(2-10) 

(2-11) 

(2-12) 


where  C  =  observation/prediction  of  interest  (e.g.,  dosage),  Cp  corresponds  to  model 
prediction,  C0  corresponds  to  observations,  C  implies  the  average,  Cc  =  standard 
deviation,  and  for  this  study  x  =  2,  5,  and  10.  We  consider  these  statistical  measures  to  be 
somewhat  standard  since  several  relatively  recent  reports  known  to  the  authors  [Refs.  2-2 
and  2-10]  make  use  of  these  measures. 

FB  and  MG  measure  the  systematic  bias  in  a  model  (relative  over-prediction  or 
under-prediction).  NMSE  and  VG  measure  the  scatter  associated  with  the  predictions 


2-18 


relative  to  observations.  Because  FB  and  NMSE  are  based  on  a  linear  scale,  they  can  be 
overly  influenced  by  infrequently  occurring  high  observed  or  predicted  dosages.12 
Alternatively,  MG  and  VG  arise  from  a  logarithmic  scale  and  can  be  influenced,  perhaps 
unduly,  by  a  few  low  observed  or  predicted  dosages.  Our  approach  is  to  compute  all  of 
these  measures  and  recognize  the  inherent  limitations  of  each.  It  is  expected  that,  taken 
together,  the  eight  measures  described  above  can  provide  good  insight  into  the  relative 
performance  of  a  given  model.13 

An  additional  issue  exists  for  the  measures  that  are  computed  on  a  logarithmic 
scale.  Situations  in  which  the  observation  or  prediction  is  exactly  0.0  cannot  be  directly 
dealt  with  -  log(0)  is  undefined.  Typical  approaches  to  this  problem  have  included 
leaving  those  observation/prediction  pairs  that  include  a  zero  out  of  the  calculation  -  an 
undoubtedly  biased  approach  -  or  somewhat  arbitrarily  setting  zeros  equal  to  some 
minimum  value.  In  general,  tests  for  sensitivity  to  the  exclusion  of  data  will  need  to  be 
performed  to  allow  for  robust  conclusions.  Of  course,  MG  and  VG  estimates  might  be 
very  sensitive  to  the  choice  of  this  minimum  value.14 

In  addition  to  the  ArcMax  and  the  CWI  dosage,  estimates  of  the  plume  width  and 
centerline  direction  were  also  derived  for  each  arc.  We  defined  the  width  of  the  plume  by 
“cutting”  equal  crosswind  integrated  dosages  from  the  sides  of  the  plume.  Given  a 
particular  fraction  of  the  crosswind  integrated  dosage  that  we  want  to  capture  inside  the 
plume  (e.g.,  0.75),  we  can  find  the  maximal  index  kieft,  and  minimal  index  kright  (from  the 
left  and  right,  respectively)  such  that: 


12  In  fact,  our  initial  observation  of  large  NMSE  values  (with  associated  very  large  estimated  confidence 
intervals)  led  us  to  carefully  re-examine  the  Prairie  Grass  field  trial  computer  files  that  had  been 
provided  to  us.  A  few  apparently  spurious  peaks  in  the  data  were  identified  and  their  origins  were 
investigated.  The  inclusion  of  these  few  peaks  (i.e.,  ArcMax  values)  resulted  in  large  NMSE  and  very 
large  associated  confidence  intervals.  As  a  result  of  this  investigation,  we  found  a  few  places  where  a 
decimal  point  had  been  misplaced  in  converting  the  “hardcopy”  of  the  Prairie  Grass  field  trial  sampler 
data  into  computer  format.  Additionally,  we  concluded  that  two  “spurious”  peaks  that  were  recorded 
in  the  hardcopy  of  the  report  were  due  to  incorrect  placement  of  the  decimal  point  by  the  authors  of 
that  report.  Appendix  K  details  these  findings  and  corrections. 

13  Perfect  agreement  with  a  set  of  observations  would  result  in  MG,  VG,  R,  FAC2,  FAC5,  and  FAC10  = 
1.0;  and  FB  and  NMSE  =  0.0. 

14  For  this  study,  we  do  not  report  point-to-point  comparisons.  That  is,  we  only  present  statistical 
measures  associated  with  single  characterizations  of  each  sampler  arc,  for  example,  ArcMax  and  CWI 
dosage.  The  values  of  ArcMax  and  CWI  dosage  for  the  valid  Prairie  Grass  field  trials  were  always 
greater  than  zero.  Therefore,  we  do  not  encounter  the  problem  of  “zeros”  in  this  study  and  can  directly 
compute  measures  such  as  MG  and  VG. 
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(2-13) 


kIefi  =  max |  k  X-=i Di  *  '—J ~  =  °-125} 

V=mi"{t|S",-D(£lz7^=o-125} 

where  D,  is  a  dosage  collected  at  sampler  i. 

Then,  we  define  width  for  this  equal-cuts-off-the-sides  method  (Wec)  as: 

=rex(krlsl,-k,J  (2-14) 

where  r  is  the  distance  in  meters  to  the  arc  and  0  is  the  angular  separation  (in  radians) 
between  the  samplers.  Wec  is  estimated  in  units  of  meters.  For  this  study,  we  considered 
widths  that  included  75  percent  of  the  CWI  dosage  associated  with  the  given  arc  (as  in 
equation  2-13). 

We  defined  the  centerline  direction  (CentDir)  in  the  following  way.  We 
considered  the  point  that  divides  the  plume  into  two  equally  weighted  plumes  (i.e.,  this  is 
a  median-based  calculation).  We  define  the  median  of  mass  (MOM)  centerline  of  the 
plume  for  any  given  arc  by  finding  the  sampler  with  index  MOM,  (smom),  such  that: 

MOM  =  minjfc  |  V'  D  -a.-V  />  j.  (2-15) 

CentDir  is  measured  in  units  of  degrees.15  Additional  comments  and  discussion 
of  alternative  methodologies  for  estimating  the  plume  width  and  direction  for  Prairie 
Grass  field  trial  data  can  be  found  in  Appendix  J  of  this  document. 

For  Wec,  we  computed  the  NMSE  and  FB.16  For  CentDir,  we  computed  the  root 
mean  square  error  (RMSE)  and  bias  as  defined  below:17 

RMSE  =  ^j(C0-Cp)2  (2-16) 

Bias  =  C~„  -~C~p  .  (2-17) 

15  Because  the  angular  separation  of  the  samplers  for  the  50-  through  400-meter  arcs  is  2  degrees,  the 
precision  with  which  CentDir  MOM  can  be  known  is  2  degrees.  For  the  800-meter  arc,  the  angular 
separation,  and,  hence  precision,  is  1  degree. 

lf)  Since  Wec  values  did  not  vary  by  large  amounts  (e.g.,  orders  of  magnitude),  we  do  not  present 
logarithm-based  measures  (i.e.,  MG  and  VG)  in  this  case. 

17  NMSE  and  FB  were  deemed  inappropriate  for  use  with  CentDir.  In  particular,  the  normalizing  feature 
of  these  measures  (i.e.,  the  denominators)  requires  an  absolute  scale  which  is  absent  in  this  measure  of 
direction.  For  example,  if  CentDir  (in  degrees)  is  measured  from  the  North,  one  would  expect  the 
NMSE  and  FB  values  to  differ  when  compared  to  measurements  done,  for  instance,  from  the  South. 
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3.  Estimation  of  Confidence  Intervals 


In  order  to  assess  the  significance  of  a  particular  result,  we  computed  confidence 
intervals  (95  and  80  percent)  for  each  measure.  For  these  calculations,  we  adopted  the 
bootstrap  (resampling)  procedures  that  have  been  applied  in  recent  air  quality  model 
evaluation  studies  [Refs.  2-15  and  2-16].  We  used  a  modified  version  of  the  Boot-Code . 18 
In  addition,  for  use  in  this  study,  the  following  modifications  were  made  to  the  BootCode 
software: 

.  The  random  number  generator  was  modified.  The  unmodified  BootCode 
requires  a  separate  file  that  contains  the  “official”  list  of  random  numbers. 
This  file  has  10,000  random  numbers,  which  are  reused  in  a  circular  fashion 
(i.e.,  10,001st  random  number  is  the  same  as  the  first,  and  the  10,002nd  is  the 
same  as  the  second).  This  random  number  generation  methodology  appears  to 
be  an  artifact  associated  with  this  software’s  development  on  personal 
computers  in  the  mid-1980s.  That  is,  such  a  procedure  was  computationally 
acceptable. 

From  a  statistical  point  of  view,  this  procedure  can  be  problematic.  To 
compute  bootstrap  confidence  intervals,  1,000  “resamples”  are  typically 
required.  Using  the  same  “random  numbers”  via  the  circular  procedure 
described  above  can  lead  to  correlation  and  confidence  intervals  that  are  not  as 
exact  as  one  might  expect  based  solely  on  the  number  of  resamplings.  To  see 
this,  consider  the  following  “worst  case-like”  situation.  Assume  that  the  input 
data  set  contains  250  points  and  that  1,000  resamplings,  each  of  size  250,  are 
completed.  This  implies  that  the  original  10,000  numbers  will  be  used  exactly 
25  times.  In  fact,  the  confidence  intervals  in  this  example  will  really  be  based 
on  40  resamplings  and  not  1,000.  Whenever  more  than  10,000  random 
numbers  are  required,  this  correlation  problem  will  exist,  although  generally  to 
a  smaller  degree  than  in  the  example  shown  above. 

Our  modification  to  the  BootCode  allows  the  random  numbers  to  be  chosen 
from  a  modem  random  number  generator,  based  on  a  Numerical  Recipes  [Ref. 
2-17]  formulation.  This  modification  refills  the  original  10,000  random 
numbers  with  a  new  set  when  the  end  of  the  list  is  reached. 

.  Additional  blocking  features  were  added.  This  modification  allows 
confidence  interval  information  to  be  output  for  each  defined  block.  For 


The  BootCode  software  (Sigma  Research  Corporation,  S.  R.  Hanna)  as  modified  by  LLNL  to  output 
FAC  5  and  FAC  10. 
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example,  if  data  are  blocked  by  range,  then  confidence  interval  information  is 
output  at  each  range.19 

.  The  percentile  method,  rather  than  the  bootstrap-t  method,  was  used  for 
confidence  interval  estimation.  The  BootCode  calculates  two  types  of 
confidence  intervals:  “robust”  and  “seductive.”  Robust  intervals  appear  to  be 
based  on  the  bootstrap-t  procedure  and  the  seductive  intervals  are  based  on  the 
percentile  method  [Ref.  2-18].  The  authors  of  the  BootCode  recommend 
using  the  robust  intervals.20  Basically  the  bootstrap-t  method  (also  known  as 
robust)  allows  one  to  estimate  the  mean  and  variance  and  then,  based  on  a 
Student’s  t-distribution,  generate  confidence  limits.  This  technique  was 
employed  instead  of  the  percentile  (or  seductive)  method  because  it  was 
known  that  the  tails  of  the  given  distribution  might  be  less  accurately  known 
than  the  mean  and  variance.  The  percentile  method  directly  uses  the  tails  of 
the  distribution  to  generate,  for  instance,  the  95th  percent  confidence  interval. 
The  concern  with  less  accurate  tails  is  that  estimates  of  confidence  intervals 
based  on  them  might  be  inaccurate  or  biased.  Hence,  the  recommendation  to 
use  the  bootstrap-t  procedure. 

However,  the  t-distribution  may  not  be  appropriate  for  the  given  statistic.  For 
example,  using  the  recommended  BootCode  procedures,  we  often  found  that 
estimates  of  the  lower  bound  limits  for  NMSE  were  negative  -  a  physically 
implausible  situation.  Similarly,  estimated  upper  limits  on  correlation,  R, 
could  be  above  1.0. 

A  simple  fix  to  this  problem  would  be  to  allow  the  user  to  choose  an 
appropriate  distribution  depending  on  the  statistic  being  examined  (e.g., 
perhaps  log-normal  in  some  cases).  The  BootCode  software  that  we  had  did 
not  easily  allow  for  this  decision.  In  addition,  considering  the  varied  potential 
users  of  this  software,  allowing  such  decisions  by  all  users  may  not  be  deemed 
“operationally  robust”  (i.e.,  some  users  might  make  incorrect  decisions). 

We  decided  simply  to  use  the  percentile  method  for  all  calculations  of 
confidence  intervals  -  essentially  to  avoid  implausible  confidence  intervals  on 
NMSE.  We  also  note  that  the  amount  of  bias  resulting  from  a  bootstrap 
procedure  can  be  estimated  by  comparing  the  bootstrap  sample  estimates  to 
the  original  point  estimate.  Furthermore,  if  substantial  bias  is  identified,  then 


19  In  addition,  a  minor  modification  to  the  BootCode  allowed  us  to  compute  approximate  confidence 
intervals  for  RMSE  and  bias. 

20  See  page  2271  of  Reference  2-2  and  page  1386  of  Reference  2-16. 
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a  straightforward  bias  correction  procedure,  known  as  the  bias-corrected  and 
accelerated  (BCa)  method,  can  be  applied.21 

Bootstrap  techniques,  similar  to  those  described  above  for  the  statistical  measures, 
were  also  used  to  assign  approximate  confidence  regions  to  MOE  2  estimates.  We  used 
our  own  resampling  routines  (vice  the  BootCode)  for  these  computations.22  For  each  trial 
prediction/observation  (and  each  individual  arc),  an  MOE  2  value,  (x,y),  was  computed. 
For  a  given  set  of  comparisons,  for  example,  all  50  meter  arcs,  the  vector  mean  of  these 
values  corresponded  to  the  MOE  2  point  estimate.  Approximate  confidence  regions  were 
computed  by  re-sampling  pairs  of  MOE  2  “observations”  (i.e.,  individual  trial  (x,y) 
vectors).  This  procedure,  resampling  in  pairs,  maintains  the  appropriate  correlation 
inherent  in  the  data  set. 

C.  TRIAL  CONDITIONS  THAT  WERE  EXAMINED 

Logicon  R&D  Associates  or  IDA  prepared  the  HP  AC  predictions  and  LLNL 
personnel  prepared  the  NARAC  predictions.  Appendix  B  of  the  paper  provides  tables 
that  describe  additional  input  details  associated  the  NARAC  and  HP  AC  predictions. 

1.  Prairie  Grass  Field  Trial  Data  Were  Collected  Along  Five  Arcs 

Some  of  the  Prairie  Grass  field  trials  were  removed  from  the  HPAC/NARAC 
comparison.  (See  Appendices  B  and  K  for  further  details.)  That  left  51  field  trials  that 
formed  a  basis  for  our  HPAC/NARAC  comparisons.  Each  Prairie  Grass  trial  contains 
data  for  five  semicircular  arcs  of  samplers  positioned  at  an  altitude  of  1.5  meters,  and  six 
towers  containing  samplers  at  various  altitudes  located  at  the  100-meter  arc.  In  this 
study,  only  samplers  located  at  semicircular  arcs  at  the  height  of  1.5  meters  were 
considered.  Figure  2-12  depicts  a  typical  Prairie  Grass  trial  result  for  the  five  arcs  that 
were  used  in  the  analyses. 

Table  2-1  lists  the  number  of  samplers  in  each  of  the  semicircular  arcs,  angular 
separation  between  samplers  in  the  arc,  and  the  Cartesian  distance  between  samplers  in 


21  See  page  184  of  Reference  2-18.  As  an  aside.  Reference  2-18,  page  160,  footnote  1  states  in  part,  “In 
practice,  however,  the  bootstrap-t  can  give  somewhat  erratic  results,  and  can  be  heavily  influenced  by  a 
few  outlying  data  points.  The  percentile  based  methods  of  the  next  two  chapters  are  more  reliable.” 

22  The  routines  used  are  based  on  in-house  modified  versions  of  spreadsheet  software  available  at 
http://web.nps.navy.mil/~orfacpag/resumePages/gaver/bootstrp.htm. 
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DOSAGE  DOSAGE 


the  arc.  For  each  arc,  a  single  value23  (for  either  the  user-oriented  MOEs  or  the 
“standard”  statistical  measures)  is  calculated,  thus  yielding  five  separate  values  per  trial. 


9 

§  200 


Q  30 


10 


ARC50 


40 

Sampler  number 

ARC 800 


100 

Sampler  number 


PG  Observation 

PG  Trial  File:  pr_>g rass_tracer_Experiment_07.txt 


Figure  2-12.  Prairie  Grass  Trial  7  as  a  Function  of  Arc:  Stability  Category  is  I24 


23  For  the  one-dimensional  MOE  (MOE  1),  this  is  a  single  number,  while  for  the  two-dimensional  MOE 
(MOE  2)  this  is  represented  by  a  pair  of  numbers. 

24  The  value  for  sampler  45  of  the  100-meter  arc  is  missing.  Dosage  units  are  in  mg-sec/m3.  Dosage 
times  a  breathing  rate  (in  m3/sec)  equals  dose  (in  mg). 
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These  values  are  combined  into  separate  groupings  that  are  then  used  in  the  comparisons. 
This  structure  of  the  Prairie  Grass  field  trials  gives  a  natural  way  of  comparing  results  by 
arc  range. 


Table  2-1.  Distance  Between  Samplers  by  Arc 


Arc 

Number  of 
Samplers 

Angular  Separation, 
degrees 

Cartesian  Separation, 
meters 

50-meter 

91 

2 

1.75 

100-meter 

91 

2 

3.49 

200-meter 

91 

2 

6.98 

400-meter 

91 

2 

13.96 

800-meter 

181 

1 

13.96 

Importantly,  the  user-oriented  MOEs  presented  in  the  previous  subsection  are 
closely  related  to  actual  area  measures.25  Thus,  there  is  a  natural  way  to  combine  MOEs 
calculated  for  each  individual  arc  into  a  single  MOE  value  that  is  associated  with  a 
particular  trial.  To  do  so,  we  associate  each  sampler  with  a  short  line  segment  centered  at 
the  sampler  location  and  having  the  same  length  as  the  inter-sampler  distance  for  that  arc. 
Then  for  AE1,  given  some  specified  threshold,  we  multiply  the  number  of  samplers  in 
each  individual  arc  used  to  define  Aov,  Afn,  and  AFP  by  the  inter-sampler  distance. 
Adding  the  values  for  the  five  arcs  together  forms  the  Aov,  Afn,  and  App  estimates  for  the 
entire  (all  arcs)  trial.  Similarly,  for  AE2,  we  combine  crosswind- integrated  dosages  for 
each  individual  arc  together  by  multiplying  them  by  the  inter- sampler  distance  for  each 
individual  arc.26 


25  In  Chapter  3,  we  provide  an  example  that  compares  MOE  values  based  on  AE1  to  those  based  on 
“actual”  areas.  In  that  case,  the  actual  areas  are  based  on  logarithmic  interpolation.  Briefly,  the 
interpolation  procedure  is  as  follows:  1)  values  obtained  at  the  samplers  are  transformed  into  log  space; 
2)  Delaney  triangulation  of  the  physical  space  between  inner  and  outer  arcs  is  performed  in  terms  of 
sampler  location;  3)  for  each  resulting  triangle,  the  data  values  within  a  triangle  are  linearly 
interpolated;  4)  the  interpolated  values  are  transformed  back  to  linear  space.  Steps  2)  and  3)  are  done 
efficiently  within  the  Interactive  Data  Language  (IDL)  software. 

1  if  dosage  at  sampler  i  >  threshold 

-6  By  defining  the  function  f(i)  =  ,  we  recognize  that,  for  AE1, 

[  0  otherwise 

A0v,  APn,  and  APP  are  integrals  over  the  sub-arcs  that  contain  samplers  with  observations  above  the 
defined  threshold.  Similarly,  AE2  and  lethality/effects  filter-based  estimates  correspond  to  integrals 
over  sub-arcs.  Hence,  there  is  a  natural  way  of  combining  arcs  at  different  ranges,  as  long  as  the 
integration  preserves  the  distance  units.  This  is  achieved  by  converting  each  integral  from  the  “number 
of  samplers”  units  to  physical  length  units  (e.g.,  meters). 
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2.  Stability  Category  Groupings 

Stability  category  assignments,  that  were  developed  by  Irwin  [Ref.  2-19]  for  the 
Prairie  Grass  trials,  were  used  to  group  trials  with  similar  characteristics  in  terms  of 
atmospheric  stability.  There  are  seven  stability  categories  ranging  from  “very  unstable” 
(category  1)  to  “very  stable”  (category  7).  Because  of  the  relatively  small  sample  sizes 
involved,  we  used  two  different  procedures  to  combine  categories  together,  as  depicted  in 
Table  2-2.  Comparing  results  obtained  by  these  categorizations  of  the  data,  allowed  for 
the  examination  of  the  statistical  spread  to  assess  the  effects  of  the  relatively  small 
sampling  sizes  for  some  groupings.  Table  2-3  lists  the  Prairie  Grass  data  sample  sizes 
that  were  used  in  the  model  comparisons. 

Table  2-2.  Stability  Category  Groupings  (SCG)  Used  in  This  Study27 


SCG  Set  1 

SCG  Set  2 

Unstable 

1,2 

1,2,3 

Neutral 

3,4,5 

4 

Stable 

6,7 

5,6,7 

3.  Thresholds  for  Sampler  Observations 

SO2  concentrations  were  estimated  in  the  following  way.  First,  SOo  was  captured 
at  the  impingers  (samplers).  These  impingers  were  later  filled  with  a  hydrogen  peroxide 
(H2O2)  solution.  This  caused  the  SO2  and  H2O2  to  form  sulfuric  acid  (H2SO4).  Next,  the 
concentration  of  sulfuric  acid  was  estimated  by  measuring  the  electrical  conductivity. 
Based  on  the  computed  sulfuric  acid  concentration,  an  estimate  of  the  original  SO2 
concentration  was  derived. 

The  reported  sampler  minimal  averaged  concentration  value  for  the  Prairie  Grass 
field  trials  is  0.005  mg/m  ,  which  corresponds  to  a  minimal  dosage  of  3  mg-sec/m  . 
However,  to  obtain  averaged  concentration  values  for  samplers  inside  the  gas  plume,  it 
was  necessary  to  estimate  the  “background”  averaged  concentrations  for  SO2.  To  do  so, 
some  samplers  were  placed  outside  of  the  limits  of  the  time-mean  gas  plume.  In  general, 
this  small  background  concentration  level  was  almost  entirely  due  to  a  trace  amount  of 
sulfuric  acid  that  had  been  added  during  the  preparation  of  the  dilute  hydrogen  peroxide 


27 


A  more  physically-based  arrangement  would  be  to  group  the  near  neutral  categories,  3  and  4,  together. 
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Table  2-3.  Sample  Sizes  Used  for  Model  Comparisons  to  Prairie  Grass  Field  Trials28 


SCG 

All  Arcs 

50-Meter 

100-Meter 

200-meter 

400-meter 

800-meter 

1 

6 

6 

6 

6 

6 

429 

2 

11 

1 03° 

11 

1031 

11 

11 

3 

9 

9 

S32 

9 

9 

9 

4 

10 

10 

10 

10 

10 

10 

5 

7 

7 

7 

7 

7 

7 

6 

6 

6 

6 

6 

6 

533 

7 

2 

2 

2 

2 

2 

2 

Total 

51 

50 

50 

50 

51 

48 

First  procedure  used  to  combine  stability  categories 

1-2 

17 

1622 

17 

1623 

17 

1521 

3-4-5 

26 

26 

2524 

26 

26 

26 

6-7 

8 

8 

8 

8 

8 

725 

Second  procedure  used  to  combine  stability  categories 

1-2-3 

26 

2522 

2524 

2523 

26 

2421 

4 

10 

10 

10 

10 

10 

10 

5-6-7 

15 

15 

15 

15 

15 

1425 

28  Sub-groupings  for  the  comparisons  include  all  arcs  (highlighted  in  pink);  sub-grouping  by  range 
(highlighted  in  red);  sub-grouping  by  all  arcs  aid  stability  group  (highlighted  in  gray);  and  sub¬ 
grouping  by  range  and  stability  group  (highlighted  in  turquoise).  Sample  sizes  highlighted  in  red  and 
turquoise  are  applicable  to  the  “standard”  statistical  measures  and  user-oriented  MOEs  that  we 
computed  while  gray  and  pink  highlighted  entries  apply  to  the  user-oriented  MOEs  only. 

-9  The  800-meter  arcs  of  trials  7  and  16  have  maximum  values  below  the  data  threshold  of  60  mg -sec/m3. 
These  two  arcs  were  included  only  for  calculations  done  with  a  cutoff  threshold  set  to  3  mg  -sec/m3. 

30  The  50-meter  arc  of  trial  62  has  a  missing  correction  factor  for  the  compensation  of  the  evaporative 
loss  of  the  impinger  solution  during  aspiration  [Ref.  2-20,  p.  201]. 

3 1  The  200-meter  arc  of  trial  50  was  replaced  by  “missing  values”  because  the  vacuum  line  to  sampler  62 
became  disconnected  during  the  run  [Ref.  2-20,  p.  80]. 

32  The  100-meter  arc  of  trial  57  was  replaced  by  “missing  values”  because  the  vacuum  line  to  sampler  47 
was  believed  to  have  become  disconnected  [Ref.  2-20,  p.  80]. 

33  The  800-meter  arc  of  trial  39  has  too  many  samplers  that  are  missing.  Of  19  samplers  that  recorded 
non-zero  values,  13  have  missing  values.  The  HP  AC  prediction  fit  completely  within  this  missing 
value  region.  In  this  case,  we  replaced  the  whole  arc  with  “missing  values.” 
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solution.  Because  of  this,  as  the  limit  of  the  sampling  technique  was  approached,  the 
uncertainty  of  the  averaged  SO2  concentration  determination  increased  rapidly.  For 
concentrations  less  than  0.1  mg/m  (60  mg-sec/m  ),  this  uncertainty  was  estimated  to  be 
25  percent.34 

The  mechanism  described  above  was  considered  the  chief  source  of  uncertainty 
associated  with  estimates  of  SO2  concentrations  and,  hence,  dosages.  For  this  reason,  we 
chose  0.1  mg/m3  (60  mg-sec/m3)  as  the  data  cutoff  threshold.35 

4.  Comment:  Value  of  Graphics  Tools 

The  majority  of  the  computer  software  used  to  manipulate  the  field  trial  and 
prediction  data,  and  calculate  the  user-oriented  MOEs,  was  written  in  Interactive  Data 
Language  (IDL)  [Ref.  2-21].  This  computer  language  provides  a  flexible  platform  for  the 
interactive  investigation  and  manipulation  of  the  data  with  an  emphasis  on  graphics.  One 
of  the  valuable  lessons  that  we  learned  early  on  was  that  the  best  way  to  analyze  or 
compare  data  quickly  is  to  find  a  suitable  format  to  graph  it.  To  that  extent,  a  prerequisite 
for  the  best  kind  of  user-oriented  MOE  (and  perhaps  also  for  a  “standard”  statistical 
measure)  is  that  it  should  be  able  to  describe  all  of  the  major  nuances  of  the  plots  that  are 
easily  discernible  by  the  human  eye.  For  this  analysis,  Appendices  C,  D,  E,  F,  G,  L,  and 
M  -  graphical  plots  of  the  observations  and  predictions  -  represent  invaluable  tools  with 
respect  to  confirming  our  interpretations  of  the  MOE  results. 


34  See  Reference  2-20,  p.  77. 

35  In  Section  A.2.d  of  Chapter  3,  we  briefly  set  the  data  cutoff  threshold  to  3  mg-sec/m3.  This  was  done 
to  allow  for  the  demonstration  (only)  of  a  lethality/effects  filter  as  applied  to  a  highly  toxic  substance. 
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CHAPTER  3 

RESULTS,  ANALYSES,  AND  DISCUSSION 


3.  RESULTS,  ANALYSES,  AND  DISCUSSION 


This  chapter  provides  the  results  of  comparisons,  done  with  user-oriented  MOEs, 
of  HP  AC  and  NARAC  predictions  to  Prairie  Grass  field  trial  observations.  This  chapter 
includes  a  few  comparisons  of  these  models  using  standard  statistical  measures. 
Additional  details  of  these  comparisons  are  given  in  Appendices  C,  D,  E,  F,  G,  H,  I,  J,  K, 
L,  and  M. 

A.  COMPARISONS  OF  HPAC  AND  NARAC  PREDICTIONS  WITH  MOEs 

This  section  is  divided  into  four  parts.  The  first  part  describes  results  based  on 
MOE  1.  The  next  part  discusses  the  2D  MOE  (MOE  2)  results.  The  third  part  provides 
analyses  and  discussion  associated  with  two  HPAC  prediction  excursions.  Finally,  two 
NARAC  excursion  prediction  comparisons  are  described. 

1.  MOE  1 

a.  Across  All  Trials 

Figure  3-1  compares  MOE  1  values  based  on  the  threshold  area  estimate  AE1  for 
HPAC  and  NARAC  predictions  for  two  different  Cfn/Cfp  values  and  a  dosage  threshold 
of  60  mg-sec/m3.  In  this  figure,  the  mean  value  and  the  50th,  80th,  and  95th  percent 
confidence  intervals  for  the  point  estimates  are  shown.1 

On  the  left,  false  positive  (i.e.,  App)  and  false  negative  (i.e.,  Afn)  are  equally 
weighted  (e.g.,  Cfn  =  CFp  =  1).  For  these  predictions  and  this  measure,  the  HPAC 
predictions  achieve  a  larger  MOE  1  value.  This  difference  is  statistically  significant.  The 
right  side  of  Figure  3-1  illustrates  the  results  when  Afn  is  weighted  at  ten  times  the 
amount  of  AFP  (i.e.,  Cfn  =  5,  Cpp  =  0.5). 2  Under  these  conditions,  the  NARAC 
predictions  achieve  the  statistically  larger  value  of  MOE  1.  The  implication  is  that,  on 


1  In  the  figure,  LCL  =  Lower  Confidence  Limit  and  UCL  =  Upper  Confidence  Limit. 

2  Perhaps  this  situation  corresponds  to  a  user  that  is  much  more  concerned  with  accidental  exposure 
relative  to  unnecessary  evacuation. 
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o 

average  at  a  threshold  dosage  of  60  mg-sec/m  ,  the  NARAC  predictions  are  more 
conservative  than  the  HP  AC  predictions  (tend  to  overpredict  relative  to  HP  AC). 3 


Figure  3-2  shows  a  similar  comparison  for  MOE  1  using  the  summed  dosages, 
AE2,  to  estimate  Aov,  Afn,  and  App.  HP  AC  and  NARAC  results  are  similar  for  the 
equally  weighted  case.  This  suggests  that  the  predictions  of  dosage  appear  consistent 
between  the  models.  The  increased  user  weighting  of  Afn  (right  side  of  Figure  3-2)  leads 
to  an  improved  MOE  1  value  for  NARAC  relative  to  HP  AC.  These  conclusions  are 
consistent  with  the  MOE  1  AE1  results  of  Figure  3-E 
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Figure  3-1.  MOE  1,  AE1,  All  Trials:  For  51  Trials  and  Two  Sets  of  User  Coefficients 


1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


AE2:  Cfn  =  1,  Cfp  =  1 


♦ 


♦  Mean 

♦  50%LCL 

♦  50%UCL 

♦  80%LCL 

♦  80%UCL 
x  95%LCL 
x  95%UCL 


1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


AE2:  Cfn  =  5,  Cfp  =  0.5 


Mean 

50%LCL 

50%UCL 

80%LCL 

80%UCL 

95%LCL 

95%UCL 


NARAC 


HPAC 


NARAC 


HPAC 


Figure  3-2.  MOE  1,  AE2,  All  Trials:  For  51  Trials  and  Two  Sets  of  User  Coefficients 


3  An  inspection  of  the  plots  deposited  in  Appendix  E  confirms  this  interpretation. 
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b.  As  a  Function  of  Range 

Figure  3-3  shows  MOE  1  comparisons  for  HP  AC  and  NARAC,  as  a  function  of 
range,  for  equal  weighting  of  false  positive  and  false  negative  areas.  Similar  behavior,  a 
decrease  in  MOE  1  value  at  the  longest  range,  is  observed  for  both  models  (and  with  both 
area  estimates  -  AE1  and  AE2). 
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Figure  3-3.  MOE  1,  AE1  and  AE2:  By  Arc  Range  (CFn  =  CFp  =  1) 


c.  As  a  Function  of  Stability  Category 

Figure  3-4  shows  the  computed  MOE  1  values  for  AE1  and  AE2  as  a  function  of 
the  stability  category  grouping  (SCG).  Individual  stability  categories  for  the  51  Prairie 
Grass  trials  used  in  this  comparison  are  based  on  Reference  3-1.  Two  independent  sets 
of  SCGs  are  shown  in  Figure  3-4  -  1,2;  3,4,5;  6,7  and  1,2,3;  4;  5, 6, 7. 4  Additional 
comparisons  of  HPAC  and  NARAC  MOE  1  estimates  for  the  different  SCGs  and  at  each 
arc  distance  (i.e.,  SCG  x  arc  range)  are  provided  in  Appendix  H. 


4  For  example,  the  SCG  labeled  “1,2”  implies  that  the  very  unstable  and  unstable  trials  were  combined. 
A  more  physically-based  arrangement  would  be  to  group  the  near  neutral  categories,  3  and  4,  together. 
The  conclusions  of  this  paper  are  not  substantially  affected  by  this  re-grouping. 
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Stable  trials  are  typically  characterized  by  plumes  with  relatively  narrow 
crosswind  widths  and  sharp  peaks.  Unstable  trials  are  represented  by  wider  crosswind 
plumes  with  lower  relative  peak  values. 

As  measured  by  MOE  1,  the  performance  of  both  HP  AC  and  NARAC  decreases 
significantly  for  the  more  stable  trials  (6,7  and  5,6,7).  For  both  models  and  for  both 
MOE  1  estimates  (i.e.,  AE1  and  AE2),  the  models  achieve  the  highest  values  for  the 
neutral  or  unstable  trials.  The  relatively  small  sample  size  associated  with  the  more 
stable  trials  (sample  size  for  SCG  6,7  =  8)  is  reflected  in  the  larger  confidence  intervals 
associated  with  the  point  estimates. 
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Figure  3-4.  MOE  1,  AE1  and  AE2:  By  Stability  Category  Grouping  (CFN  =  CFp  =  1) 

The  next  section  invokes  the  two-dimensional  MOE  and  further  develops  the 
differences  between  HPAC  and  NARAC  predictions  as  a  function  of  SCG. 

2.  MOE  2 


a.  Across  All  Trials 

Figure  3-5  illustrates  our  estimates  of  MOE  2.  Each  point  shown  in  the  figure 
represents  a  bootstrap  estimate  of  MOE  2.  A  total  of  1,000  bootstrap  resamplings  were 
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computed  and  950  points  are  shown  for  each  estimate.  The  area  covered  by  these  950 
points  corresponds  to  the  approximate  95th  percent  confidence  region.  The  point 
estimates  for  MOE  2  reside  at  approximately  the  center  of  the  colored  confidence 
regions. 
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Figure  3-5.  MOE  2,  AE1  and  AE2,  All  Trials:  For  51  Prairie  Grass  Trials 

Using  the  critical  threshold  area  estimate,  the  HPAC  MOE  2  AE1  value  of  (0.80, 
0.85)  lies  somewhat  closer  than  the  corresponding  NARAC  estimate  (0.94,  0.62)  to  the 
best  possible  value  of  (1,1). 5  According  to  the  left  side  chart  in  Figure  3-5,  the  NARAC 
predictions  exhibit  a  smaller  false  negative  fraction  relative  to  the  HPAC  predictions. 
The  HPAC  predictions  show  a  smaller  false  positive  fraction  relative  to  the 
corresponding  NARAC  predictions.  These  conclusions  are  statistically  significant  (i.e., 
the  95th  percent  confidence  regions  are  completely  separated).  The  AE2  estimates,  in 
which  the  “areas”  are  based  on  the  summed  crosswind  dosages,  show  similar  results 


5  As  briefly  mentioned  in  Chapter  2  (footnote  24),  we  also  computed  MOE  2  using  interpolated  areas 
(e.g.,  in  m2)  to  estimate  APP,  AFN,  and  Aov.  There  was  very  little  change  in  the  estimates  and 
confidence  regions  for  MOE  2  based  on  interpolated  areas  relative  to  AE1  -  hence,  our  occasional 
reference  to  AE1  as  a  natural  area  estimate  (at  least  for  the  Prairie  Grass  field  trial  data).  The  point 
estimates  for  HPAC  MOE  2  based  on  AE1  and  interpolated  area  were  (0.796,  0.850)  and  (0.818, 
0.848),  respectively.  Similarly,  for  NARAC,  the  estimates  were  (0.935,  0.616)  and  (0.937,  0.615), 
respectively.  A  future  IDA  paper  will  document,  compare,  and  contrast  a  few  area  interpolation 
schemes. 
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(right  side  chart  in  Figure  3-5)  but  with  the  NARAC  estimate  (0.74,  0.71)  somewhat 
closer  than  the  HP  AC  estimate  (0.60,  0.79)  to  the  best  possible  value  (1,1). 


b.  As  a  Function  of  Range 


Figure  3-6  compares  MOE  2  confidence  regions  as  a  function  of  arc  range.  There 
is  evidence  that  both  models’  performance  degrades  at  the  longer  ranges.  For  both 
models  and  for  both  AE1  and  AE2  estimates,  the  results  for  the  800-meter  arc  are 
significantly  degraded  relative  to  the  50-meter  arc.  For  the  critical  threshold  area 
estimate,  MOE  2  AE1,  the  chart  on  the  left  side  of  Figure  3-6  illustrates  that  as  the  range 
increases,  relative  to  the  50-meter  arc,  the  HP  AC  prediction  tends  toward  increased  false 
positive  with  minimal  increases  in  false  negative.  For  the  NARAC  predictions,  as  the 
range  increases  the  results  tend  toward  increased  false  negative  with  at  first  minor 
decreases  in  false  positive  fraction.  The  50-  and  800-meter  arcs,  for  NARAC  MOE  2 
AE1,  have  similar  false  positive  fractions. 
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Figure  3-6.  MOE  2,  AE1  and  AE2:  As  a  Function  of  Arc  Range 

For  these  data,  MOE  2  AE2  provides  complete  separation  of  confidence  regions, 
indicating  statistically  significant  differences,  between  the  results  for  most  arcs.  For 
example,  for  the  NARAC  MOE  2  AE2  estimates,  the  95th  percent  confidence  regions  are 
generally  separate  (with  the  exceptions  of  the  507100-meter,  1007200-meter,  and  200- 
/400-meter  comparisons).  That  is,  of  the  ten  possible  arc  range  comparisons  for  NARAC 
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predictions  of  the  Prairie  Grass  trials,  MOE  AE2  reveals  statistically  significant 
differences  in  seven  instances. 

For  the  HP  AC  predictions,  a  similar  examination  of  the  AE2-based  MOE  2  95th 
percent  confidence  regions,  suggests  that  the  800-meter  arc  result  is  significantly 
different  from  the  other  four  arc  range  results.  Thus,  four  of  the  ten  possible  arc  range 
comparisons  for  the  HPAC  predictions  suggest  statistically  significant  differences  -  the 
four  involving  the  800-meter  arc. 

We  also  considered  some  standard  statistical  measures.  Figure  1-7  in  Appendix  I 
presents  results  based  on  standard  statistical  measures  (FB  and  NMSE)  for  CWI  dosage.6 
Figure  1-7  shows  that  no  statistically  significant  separation  or  trend  can  be  detected  for 
NMSE  as  a  function  of  arc  range  for  the  NARAC  or  HPAC  predictions.  FB  for  CWI 
dosage  (also  Figure  1-7)  shows  a  trend  of  increased  NARAC  over-prediction  at  the  longer 
arc  ranges  (consistent  with  the  increasing  false  positive  fraction  shown  for  MOE  2  AE2). 
However,  none  of  the  NARAC  FB  values  by  range  for  CWI  dosage  are  different  in  a 
statistically  significant  sense.  Similarly,  just  one  of  the  HPAC  comparisons  results  in  a 
statistically  significant  conclusion  (the  100-/800-meter  comparison,  i.e.,  the  95th  percent 
confidence  regions  are  completely  separate). 

Standard  statistical  measures  that  consider  the  logarithms,  MG  and  VG,  de- 
emphasize  the  largest  concentration  values,  by  increasing  the  weight  of  the  contribution 
of  the  outer  arcs  and  the  tails  of  the  distribution  on  the  inner  arcs,  relative  to  statistical 
measures  that  do  not  use  logarithms.  For  MG  values,  Figure  1-8  shows  a  trend  toward 
increasing  over-prediction  with  increased  range  for  NARAC  CWI  dosage  with  six  (of  ten 
possible)  arc  range  comparisons  different  in  a  statistically  significant  sense.  That  is,  the 
NARAC  MG  CWI  dosage  results  for  all  four  comparisons  involving  the  800-meter  arc 
and  the  50-  and  100-meter  comparisons  to  the  400-meter  arc  are  significantly  different. 
Figure  1-8  also  shows  VG  values  for  NARAC  CWI  dosage.  In  this  case,  four 
comparisons  suggest  statistically  significant  differences  -100-/800-meter,  200-/800- 
meter,  400-/800-meter,  and  100-/400-meter. 

As  previously  mentioned  for  HPAC,  both  MOE  2  AE1  and  MOE  2  AE2  resolve 
statistically  significant  performance  differences  between  the  50-  and  800-meter  arcs. 


6  The  calculations  of  MOE  2  AE2  and  CWI  dosage  involve  the  summing  of  dosages  across  an  arc. 
However,  unlike  MOE  2  AE2,  the  standard  statistics  for  CWI  dosage  do  not  include  errors  in  the 
spatial  position  of  observed  and  predicted  plumes.  Therefore,  the  CWI  standard  statistical  measures 
and  the  AE2 -based  MOE  assess  somewhat  different  aspects  of  model  performance. 
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Furthermore,  for  MOE  AE2,  the  performance  at  the  800-meter  arc  is  completely 
separated  from  all  of  the  other  arcs.  The  statistical  measures  of  FB,  NMSE,  and  VG 
show  no  statistically  significant  performance  differences  between  any  arc  ranges  for 
HPAC  CWI  dosage.  For  the  HPAC  CWI  dosage  parameter,  statistically  significant 
differences  in  MG  (Figure  1-8)  are  apparent  for  the  50-/1 00-meter,  50-/800-meter, 
100-/800-meter,  100-/400- meter,  and  200-/800-meter  comparisons. 

We  also  examined  statistical  results  based  on  ArcMax  (Figures  1-1  and  1-2)  and 
found  only  a  few  significant  differences  between  arc  ranges  for  FB,  NMSE,  MG,  and 
VG.  Table  3-1  summarizes  the  results  in  terms  of  the  number  of  arc  range  comparisons, 
out  of  ten  possible,  that  led  to  statistically  significant  differences  (i.e.,  the  95th  percent 
confidence  regions  or  intervals  were  completely  separated).  The  standard  statistical 
measure  results  are  highlighted  in  red  and  the  MOE  2  results  are  highlighted  in  blue. 

Table  3-1.  Number  of  Arc  Range  Comparison  Differences  That  Are  Statistically  Resolved 


Derived 

Value 

Measure 

NARAC 

Predictions 

HPAC 

Predictions 

ArcMax 

FB 

0 

0 

NMSE 

3 

0 

MG 

4 

0 

VG 

0 

0 

CWI  Dosage 

FB 

0 

0 

NMSE 

0 

0 

MG 

6 

5 

VG 

4 

0 

AE1 -Based 

MOE  2 

2 

3 

AE2-Based 

MOE  2 

7 

4 

Of  the  measures  shown  in  Table  3-1,  MOE  2  AE2  and  MG  CWI  dosage  show  the 
most  statistically  significant  differences  in  model  performance  as  a  function  of  arc  range 
between  50  and  800  meters. 

We  also  considered  comparisons  at  each  range  between  models  (i.e.,  HPAC  50- 
meter  arc  versus  NARAC  50-meter  arc,  HPAC  100-meter  arc  versus  NARAC  100-meter 
arc  and  so  on).  There  are  five  possible  comparisons  between  models  as  a  function  of  arc 
range.  Table  3-2  lists  the  number  of  statistically  resolvable  differences  between  models 
(out  of  a  possible  5).  By  “statistically  resolvable,”  we  mean  that  the  95th  percent 
confidence  regions  or  intervals  are  completely  separate. 
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As  can  be  seen  in  Table  3-2,  the  MOEs  (MOE  2  AE1  and  AE2)  perform  the  best 
with  respect  to  detecting  differences  between  models.  The  MG  CWI  dosage  measure 
resolves  differences  between  the  models  at  all  but  the  50-meter  arc  range  (Figure  1-7). 

Table  3-2.  Number  of  Arc  Range  Comparison  Differences  Between  Models  That  Are 

Statistically  Resolved 


Derived 

Value 

Measure 

Number 

ArcMax 

FB 

1 

NMSE 

0 

MG 

1 

VG 

0 

CWI  Dosage 

FB 

3 

NMSE 

0 

MG 

4 

VG 

2 

AE1 -Based 

MOE  2 

5 

AE2-Based 

MOE  2 

5 

Later  in  this  chapter  we  describe  results  for  a  direct  measure  of  the  plume  width, 
Wec.  This  parameter  is  developed  and  described  in  Chapter  2  and  Appendix  J. 
Examination  of  FB(Wec)  led  to  statistically  significant  differences  between  the  models  at 
all  5  arc  ranges  (Figure  3- 17). 7 

c.  As  a  Function  of  Stability  Category 

Figure  3-7  provides  MOE  2  results  for  two  sets  of  stability  category  groupings. 
The  top  two  charts  of  Figure  3-7  present  AE1  results  and  the  bottom  two  charts  provide 
AE2  results.  The  charts  on  the  left  side  show  results  for  SCGs  1,2,3;  4;  and  5,6,7.  The 
charts  on  the  right  present  results  for  SCGs  1,2;  3,4,5;  and  6,7. 

First,  we  consider  the  AE1  estimates  (top  charts).  With  respect  to  HP  AC  and 
NARAC  comparisons,  the  MOE  2  results  are  most  similar  for  the  trials  conducted  during 
the  more  unstable  conditions  (1,2  and  1,2,3)  with  the  NARAC  predictions  generating 
slightly  larger  false  positive  fractions  and  slightly  smaller  false  negative  fractions.  This 
trend  is  greatly  magnified  for  the  more  neutral  trials  (3,4,5  and  4).  The  biggest  separation 


7  No  statistically  significant  differences  between  ranges  for  a  given  set  of  model  predictions  are  seen  for 
Wpr. 
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Figure  3-7.  MOE  2,  AE1  and  AE2:  As  a  Function  of  Stability  Category  Grouping 

is  associated  with  the  relatively  stable  trials  (5,6,7  and  6,7). 8  For  both  the  neutral  and  the 
stable  conditions,  the  NARAC  predictions  generate  relatively  little  fraction  associated 
with  false  negative  (i.e.,  along  the  x-axis  the  confidence  region  lies  close  to  1).  The 
HPAC  predictions  of  the  neutral  and  stable  trials  lead  to  larger  false  negative  but  smaller 
false  positive  fractions.  The  HPAC  and  NARAC  SCG-associated  results  appear  to  be  the 
“mirror  image”  of  one  another  (with  the  symmetry  axis  lying  on  the  diagonal  that 


8  The  larger  confidence  regions  associated  with  the  stable  trials  labeled  5,6,7  and  6,7  are  due,  in  part,  to 
the  correspondingly  smaller  sample  size. 
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approximately  cuts  between  the  two  unstable  confidence  regions).  Both  HP  AC  and 
NARAC  appear  to  perform  the  best,  with  respect  to  being  closest  to  the  best-case  (1,1) 
value,  for  the  relatively  unstable  trials.  Similarly,  both  models  performed  worst,  as 
measured  by  MOE  2,  for  the  trials  that  were  conducted  under  the  more  stable  conditions. 

The  bottom  two  charts  show  almost  complete  separation  of  confidence  regions  as 
a  function  of  model  used  and  SCG.  As  was  true  for  MOE  2  AE1,  MOE  2  AE2  implies 
higher  false  negative  fractions  for  HP  AC  predictions  relative  to  NARAC.  For  the 
relatively  neutral  and  unstable  trials  the  HP  AC  predictions  have  lower  false  positive 
fractions  than  the  NARAC  predictions.  For  the  stable  trials,  HPAC  and  NARAC  show 
similar  false  positive  fractions  for  MOE  2  AE2.  Both  HPAC  and  NARAC  appear  to 
perform  the  best,  with  respect  to  being  closest  to  the  best-case  (1,1)  value  for  MOE  2 
AE2,  for  the  relatively  unstable  trials.  Similarly,  both  models  performed  worst,  as 
measured  by  MOE  2  AE2,  for  the  trials  that  were  conducted  under  the  more  stable 
conditions. 

With  respect  to  performance  comparisons  of  the  sort  described  in  Figure  2-5,  the 
differences  between  HPAC  and  NARAC  for  each  of  the  stability  category  groupings  and 
for  AE1-  and  AE2-based  MOE  2  estimates  would  be  considered  “not  decisive.”  That  is, 
there  is  no  case  where  one  model  appears  to  have  the  clear  advantage  -  statistically 
significant  decreased  false  positive  and  decreased  false  negative  fractions  relative  to  the 
other  model. 

Investigation  showed  that  the  large  somewhat  elliptical  shape  associated  with 
HPAC  stable-condition  prediction  confidence  regions  is  due,  in  part,  to  slight  errors  in 
the  transport  direction.  For  the  stable  cases,  the  HPAC  plumes  are  very  narrow  and  there 
are  a  few  trials  for  which  the  HPAC  prediction  is  about  the  right  size  and  shape  but  the 
centerline  appears  off  by  a  few  degrees  (perhaps  due  to  limited  wind  direction 
information).9  If  the  direction  of  the  prediction  and  the  actual  plume  is  identical,  the 
overlap  is  maximized  and  the  false  positive  and  false  negative  areas  are  minimized. 
Therefore,  improved  wind  direction  information  can  improve  both  the  false  positive  and 
false  negative  aspects  of  MOE  2.  Similarly,  somewhat  incorrect  wind  direction 
information  should  lead  to  degraded  MOE  2  values  along  both  the  false  positive  and  false 
negative  axes.  However,  for  a  very  narrow  plume,  even  slight  errors  in  direction  can 
greatly  increase  the  false  positive  and  false  negative  areas  at  the  expense  of  overlap. 


9  See,  for  example.  Appendix  C  Figures  C-27a  and  C-47a. 
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Therefore,  transport  or  perhaps  wind  direction  errors,  can  explain  the  elliptical  shape  and 
its  orientation  (i.e.,  major  axis  along  the  (0,0)  to  (1,1)  line)  shown  by  some  of  the 
confidence  regions. 

The  above  consideration,  transport  errors,  describes  a  mechanism  that  can 
correlate  the  two  axes  of  MOE  2.  For  predictions  that  have  perfect  wind  direction 
information  and  hence  accurate  transport  directions,  another  feature  -  the  relative 
predicted  and  actual  plume  width  -  could  lead  to  a  correlation.  In  this  case,  predictions 
that  are  too  wide  lower  false  negative  and  raise  false  positive  values.  Similarly, 
predictions  that  are  too  narrow  increase  false  negative  areas  at  the  expense  of  false 
positive  areas.  Such  a  mechanism  would  likely  lead  to  an  elliptical  confidence  region 
whose  major  axis  would  be  oriented  along  the  (1,0)  to  (0,1)  line. 

d.  Lethality/Effects  Filter  Results 

Figure  3-8  presents  some  sample  results  using  a  lethality/effects  filter  (as 
described  in  Chapter  2)  in  lieu  of  AE1  and  AE2.  The  charts  on  the  left  side  of  Figure  3-8 
use  a  probit  curve  with  FD50  =  100  mg-sec/m  and  a  slope  of  6.  The  charts  on  the  right 
side  of  Figure  3-8  use  a  probit  curve  with  FD50  =  5,000  mg-sec/m1  and  a  slope  of  12. 10 

Interestingly,  a  quick  comparison  of  Figures  3-7  and  3-8  reveals  that  they  are 
quite  similar  and  appear  to  convey  almost  identical  information.  That  is,  the  AE1  charts 
of  Figure  3-7  correspond  to  the  FD50  =  100  charts  of  Figure  3-8.  Fikewise,  the  AE2  and 
FD50  =  5,000  charts  appear  similar. 

This  relationship  should  not  be  a  surprise.  For  example,  for  very  toxic  substances 
(or  low  threshold  values  of  interest),  the  filter  simply  computes  a  high  fraction  of 
lethality/effects  wherever  there  is  essentially  any  false  negative  dosage.  For  a  uniformly 
distributed  exposed  population,  this  scales  (they  are  related  by  a  constant)  directly  to  the 
critical  threshold  area  (AE1)  value  for  Afn-  A  similar  comment  applies  to  the  false 
positive  and  overlap  areas.  Therefore,  MOE  2  values  computed  with  a  very  toxic 


The  results  in  this  section  represent  a  demonstration  of  the  use  of  lethality/effects  filters.  In  fact, 
lethality  and  effects  models  may  be  subject  to  poorly  known  thresholds  and  probit  slopes  (assuming  a 
probit  model).  There  may  be  uncertainties  in  the  knowledge  of  the  source  purity  and  unknown 
correction  factors  for  applying  data  to  a  general  population  versus  military  personnel.  Determining 
actual  casualty  levels  for  a  specific  scenario  would  also  need  to  include  factors  such  as  the  means  of 
exposure  (respiratory  or  percutaneous),  other  parameters  relating  to  that  exposure  (e.g.,  breathing  rate 
and  the  effect  of  clothing),  and  the  specification  of  a  particular  source  material.  We  note  here  that,  a 
priori ,  there  is  no  reason  that  a  more  detailed  lethality  or  effects  model  could  not  be  used  within  the 
above  lethality/effects-filter  framework. 
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lethality/effects  filter  lead  to  results  comparable  to  AEl-based  calculations.  A  similar 
conclusion  would  be  attained  if  one  considered  simply  a  threshold  effect  for  a  chemical 
agent  (e.g.,  the  onset  of  ocular  or  nasal  irritation  for  a  nerve  agent). 


By  SCG  (LD50  =  100,  Slope  =  6) 
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Figure  3-8.  MOE  2,  Lethality/Effects  Filter:  As  a  Function  of  Stability  Category  Grouping 

The  relationship  between  AE2-based  MOE  2  values  and  estimates  based  on  less 
toxic  filters  (e.g.,  LD5o  =  5,000  vice  100)  is  somewhat  more  complicated.  There  is  a 

a 

regime  of  toxicity,  for  example,  thousands  of  mg-sec/m  ,  where  the  lethality/effects- 
based  calculations  lead  to  results  very  similar  to  the  AE2-based  estimates  (based  on  the 
Prairie  Grass  field  trial  observations).  In  this  regime,  false  negative  dosages,  for 
instance,  appear  to  be  nearly  linearly  related  to  additional  unpredicted  exposures  that  lead 
to  death  or  some  effect.  Therefore,  the  computed  areas  for  the  lethality/effects  filter  and 
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AE2  computations  are  related  by  a  constant.  However,  there  must  exist  some  level  of 
toxicity  where  essentially  none  of  the  material  causes  any  effect  (or  perhaps  just  the  peak 
along  the  centerline  causes  an  effect).  Near  this  low  level  of  toxicity,  the  relationship 
between  AE2-based  and  lethality/effects  filter-based  MOE  2  computations  breaks  down. 

Practically,  the  lethality/effects  filter  represents  a  tunable  dial  that  relates  AE1  to 
AE2.  In  addition,  if  probit  slopes  or  other  effects  models  associated  with  agents  of 
interest  are  used,  this  filter  places  MOE  2  results  in  an  appropriate  operational  context  - 
that  is,  fractions  of  affected  populations  are  considered. 

The  MOE  1  and  MOE  2  results  presented  to  this  point  involve  an  initial  threshold 
setting  of  60  mg-sec/m3.11  We  also  computed  these  MOEs  using  a  threshold  value  of  3 

a 

mg-sec/m  .  Overall,  the  HPAC/NARAC  comparative  results  were  substantially  the  same 
for  the  two  threshold  values.  However,  this  lower  threshold  value  allowed  us  to 
demonstrate  the  lethality/effects  filter  results  using  probit  curves  that  modeled  much 
more  toxic  agents.  For  example,  we  examined  LD50  values  of  3  and  10  mg-sec/m  as 
shown  in  Figure  3-9.  As  expected  the  results  appear  similar  to  those  presented  for  MOE 
2  AE1.  We  conclude  that  the  above  lethality/effects  filter  can  be  used  to  compute  MOE 
values,  and  that  this  “tunable  dial”  not  only  connects  AE1  to  AE2,  but  more  importantly, 
can  relate  the  goodness  of  a  prediction  for  agents  of  greatly  varying  toxicity  (e.g., 
biological  versus  chemical  agents  or  threshold  versus  lethal  effects  for  chemical  agents). 


Figure  3-9.  MOE  2,  Demonstration  of  Highly  Toxic  Material  Lethality/Effects  Filter  (at 
Threshold  =  3  mg-sec/m3):  As  a  Function  of  Stability  Category  Grouping 


1 1  See  Chapter  2,  Section  C.3  for  an  explanation  of  this  choice. 
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3.  HPAC  Predictions:  Excursions 


This  section  describes  the  results  of  comparisons  of  the  baseline  HPAC 
predictions  of  Prairie  Grass  trials  to  two  excursions.  First,  we  reran  the  HPAC 
predictions  using  a  defined  deposition  velocity.12  This  allowed  the  prediction  to  consider 
surface  deposition  of  some  of  the  SCF. 

In  the  second  excursion,  the  HPAC  input  parameter  labeled  uu(calm)  was 
changed  from  0.0  to  0.25.  Essentially,  this  parameter  sets  a  lower  limit  on  the  plume 
dispersion  during  the  lightest  wind  conditions.  For  the  Prairie  Grass  field  trials  that  were 
conducted  over  flat,  mowed  fields  in  Nebraska,  these  lightest  wind  conditions  were 
typically  associated  with  the  most  stable  conditions.  The  baseline  uu(calm)  value,  0.0, 
was  suggested  just  prior  to  this  study  by  the  SCIPUFF  developer.13  The  uu(calm)  value 
of  0.25  corresponds  to  the  nominal  HPAC  default  value. 

a.  With  and  Without  Surface  Deposition 

Including  surface  deposition  should  lead  to  the  removal  of  some  SCT,  although 
perhaps  only  a  small  amount  of  material.  Therefore,  with  respect  to  using  the  summed 
dosages  (AE2)  to  estimate  AFP,  Afn  and  A0v  for  MOE  2,  we  expect  that  the  decreases  in 
predicted  material  would,  overall,  lead  to  decreases  in  false  positive  and  increases  in  false 
negative  fractions  relative  to  the  “  no  deposition”  predictions.  However,  if  deposition  is  a 
physically  important  effect  for  the  Prairie  Grass  field  trials  and  it  is  well  modeled,  then 
the  “deposition”  results  should  lead  to  reduced  false  positive  and  reduced  (or  perhaps 
only  slightly  increased)  false  negative  fractions  relative  to  the  “no  deposition” 
predictions. 

Figure  3-10  compares  two  sets  of  HPAC  predictions  using  MOE  2  AE2  -  with 
and  without  surface  deposition  included  in  the  computation.  In  addition  to  a  comparison 
of  results  with  all  5 1  trials,  this  figure  also  presents  results  as  a  function  of  arc  range  and 
SCG. 

In  all  cases,  the  inclusion  of  surface  deposition  in  the  HPAC  prediction  (the  blue 
confidence  regions  in  Figure  3-10)  led  to  slightly  increased  false  negative  and  slightly 
decreased  false  positive  areas.  For  any  individual  comparison  (e.g.,  400-meter  arc),  the 
confidence  regions  for  the  two  MOE  2  estimates  have  considerable  overlap.  The 


12  See  Appendix  B  for  additional  details. 

13  This  recommendation  was  made  by  Ian  Sykes  (ARAP)  after  a  brief  review  of  initial  comparative 
results  (on  13  July  2000). 
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difference  in  MOE  2  values  for  the  “deposition”  and  “no  deposition”  predictions  are  quite 
small,  and  not  necessarily  statistically  significant.  However,  with  respect  to  App  and  Afn, 
the  same  differences  are  seen  for  all  trials  and  for  all  conditions  examined  (all  ranges  and 
all  SCGs). 

We  confirmed  that  this  trend  was  not  simply  due  to  the  statistical  fluctuations 
associated  with  our  bootstrap  sample  size  -  1,000.  To  do  this,  we  simply  re-estimated  the 
confidence  regions  associated  with  each  of  the  two  sets  of  HP  AC  predictions  (i.e.,  with 
and  without  a  surface  deposition  mechanism  included).  The  difference  in  confidence 
region  location  between  runs  for  the  same  HPAC  set  of  predictions  was  much  less  than 
the  differences  between  sets  of  predictions.  That  is,  the  re-estimated  regions  fell 
approximately  on  top  of  the  original  estimates  of  the  confidence  regions. 

Next,  we  considered  the  following.  In  all  cases,  we  confirmed  that  the  point 
estimates  for  MOE  AE2  for  the  “with  deposition”  predictions  led  to  increased  false 
negative  and  decreased  false  positive  relative  to  the  “without  deposition”  trials.  This 
“diagonal  movement”  of  the  HPAC  MOE  2  estimates  might  be  due  to  random 
fluctuations  (because  of  the  relatively  small  number  of  trials  upon  which  the  MOE  2  AE2 
point  estimates  were  based).  In  this  case,  one  would  consider  four  equally  likely 
directions  for  the  “with  deposition”  trials  to  have  moved  -  increased  ApN/increased  App, 
increased  ApN/decreased  AFP,  decreased  ApN/increased  AFp,  and  decreased  AFN/decreased 
App.  However,  for  all  five  arc  distances  and  for  all  six  stability  category  groupings  the 
predictions  done  “with  deposition”  led  to  increased  false  negative  and  decreased  false 
positive.  Given  the  equally  likely  multinomial  choice-of-four  described  above,  one  can 
compute  the  likelihood  that  all  three  independent  SCGs  (e.g.,  1,2;  3,4,5;  and  6,7)  would 
lead  to  the  same  choice  that  we  previously  hypothesized.  For  this  situation  one  expects  a 
1  in  64  (4  )  chance  (i.e.,  significance  level  a  =  0.016)  of  observing  all  three  independent 
SCGs  with  the  same  choice  (given  it  is  due  to  random  fluctuations).  If  one  considered 
the  five  arc  ranges  as  independent  samples,  a  similar  procedure  leads  to  a  1  in  1,024  (45) 
chance  (a  <  0.001).  Thus,  at  the  a  =  0.05  significance  level,  we  can  reject  the  null 
hypothesis  that  the  consistent  trends  that  we  detect  are  due  solely  to  random  fluctuations. 

We  note,  however,  that  the  magnitudes  of  the  MOE  2  differences  between  the 
“with  deposition”  and  “without  deposition”  trials  are  smaller  than  the  uncertainty 
associated  with  the  individual  point  estimates.  An  inspection  of  the  comparative  figures 
of  Appendix  F  shows  just  how  small  the  differences  in  HPAC  “deposition”  and  “no 
deposition”  Prairie  Grass  predictions  are. 
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Figure  3-10.  MOE  2,  AE2:  HPAC  Predictions  With  and  Without  Surface  Deposition 

At  this  point,  we  might  ask  if  the  decrease  in  false  positive  fraction  is  greater  than 
the  increase  in  false  negative  fraction  associated  with  the  inclusion  of  surface  deposition. 
In  fact,  MOE  1,  with  equal  weightings  for  Afn,  Afp,  and  Aov  (he.,  Cfn  =  CFp  =  1.0), 
allows  for  exactly  this  assessment.  Recall  Figure  2-6. 

Table  3-3  compares  MOE  1  AE2  values  for  HPAC  predictions  done  with  and 
without  surface  deposition.  The  numbers  in  parentheses  correspond  to  the  95th  percent 
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confidence  intervals  associated  with  the  given  point  estimate.  The  larger  MOE  1  AE2 
value  for  each  comparison  is  highlighted  in  blue  boldfaced  text. 

Table  3-3.  MOE  1  AE2  Comparisons  for  HPAC  Predictions  of  Prairie  Grass  Field  Trials: 

With  and  Without  Surface  Deposition 


Trial  Type 

With 

Without 

All 

0.532 

(0.487- 

0.573) 

0.528 

(0.483- 

0.571) 

Range  = 

50  m 

0.566 

(0.513- 

0.610) 

0.564 

(0.512- 

0.609) 

Range  = 

100  m 

0.530 

(0.483- 

0.568) 

0.523 

(0.480- 

0.561) 

Range  = 

200  m 

0.499 

(0.453- 

0.540) 

0.494 

(0.449  - 

0.538) 

Range  = 

400  m 

0.481 

(0.427- 

0.529) 

0.476 

(0.425- 

0.524) 

Range  = 

800  m 

0.357 

(0.304- 

0.412) 

0.357 

(0.302- 

0.406) 

SCG  = 

=  1,2 

0.610 

(0.556- 

0.651) 

0.604 

(0.550- 

0.645) 

SCG  = 

1,2,3 

0.617 

(0.581  - 

0.646) 

0.612 

(0.575- 

0.639) 

SCG  = 

3,4,5 

0.541 

(0.481  - 

0.589) 

0.536 

(0.482- 

0.584) 

SCG 

=4 

0.504 

(0.436- 

0.561) 

0.501 

(0.437- 

0.555) 

SCG  = 

5,6,7 

0.403 

(0.314- 

0.500) 

0.403 

(0.308- 

0.500) 

SCG  = 

6,7 

0.335 

(0.230- 

0.458) 

0.341 

(0.238- 

0.458) 

For  most  conditions  shown  in  the  Table  3-3,  the  “with  deposition”  predictions  led 
to  marginally  better  MOE  1  AE2  values  (indicating  associated  larger  decreases  in  false 
positive  fraction  than  the  corresponding  increases  in  false  negative  fraction).  The 
exceptions  to  the  above  statement  are  the  results  for  the  800-meter  arc  and  for  the  more 
stable  trials.  In  fact,  for  the  small  sample  size  SCG  =  6,7  trials  (sample  size  =  8),  the 
“without  deposition”  MOE  1  AE2  value  is  slightly  larger  than  the  “with  deposition” 
result.  Inspection  of  Figures  F-24  (trial  32,  SCG  =  6),  F-47  (trial  58,  SCG  =  6),  and  F-48 
(trial  59,  SCG  =  5), 14  suggest  a  possible  cause  of  this  result.  In  each  of  these  cases,  the 
baseline  HPAC  prediction  (“without  deposition”)  led  to  an  under-prediction  of  a  very 
narrow  peak.  The  “with  deposition”  predictions  simply  removed  material  from  the 
predicted  peak  (particularly  at  the  800-meter  arc),  therefore  leading  to  a  larger  AE2-based 
false  negative  fraction  or  area. 

The  above  mechanism  (perhaps  active  in  just  a  few  of  the  trials)  would  result  in  a 
greater  increase  in  false  negative  fraction  than  decrease  in  false  positive  fraction,  on 


I4  Page  F-l  describes  the  meanings  of  the  various  shadings  in  the  Appendix  F  figures. 
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average  (i.e.,  for  the  MOE  1  AE2  value  which  is  the  vector  mean  for  all  trials  being 
considered).  This  is  exactly  what  is  implied  when  considering  both  MOE  2  AE2  and 
MOE  1  AE2  together  for  the  “with  deposition”  and  “without  deposition”  HPAC 
predictions  of  the  more  stable  trials. 

For  all  of  the  differences  shown  in  Table  3-3  the  95th  percent  confidence  intervals 
overlap  (as  can  be  seen  in  the  table).  We  also  inspected  the  80th  percent  confidence 
intervals  and  found  the  same  result  (i.e.,  in  all  cases  the  intervals  overlap).  Thus,  the 
differences  between  individual  comparisons  shown  in  Table  3-3  are  not  considered 
statistically  significant.  Therefore,  although  the  trend  suggested  by  MOE  1  AE2  is 
toward  improvement  with  the  inclusion  of  a  surface  deposition  mechanism,  with  the 
exceptions  of  the  800-meter  arc  and  the  more  stable  trails,  we  consider  these  results 
marginal  at  best. 

In  conclusion,  although  differences  were  detected,  the  addition  of  a  deposition 
mechanism  to  the  modeling  did  not  lead  to  a  clear  improvement  in  the  MOE  2  value  for 
the  HPAC  predictions.  Furthermore,  inspection  of  MOE  1  AE2  values,  with  the 
assumption  of  equal  weighting  of  false  positive  and  false  negative  fractions,  suggested,  at 
best,  marginal  overall  (MOE  1  AE2  (All  Trials)  =  0.532  versus  0.528)  improvement  with 
the  inclusion  of  a  surface  deposition  mechanism. 

b.  With  Different  Values  Defining  Dispersion  for  the  Lightest  Wind  Conditions 

Figure  3-11  provides  the  confidence  regions  associated  with  MOE  2  AE1  and 
AE2  estimates  for  two  sets  of  HPAC  predictions  with  differing  values  of  uu(calm).  This 
figure  compares  results  for  all  trials  and  as  a  function  of  SCG.  For  the  trials  conducted 
during  neutral  and  unstable  conditions,  the  two  types  of  HPAC  predictions  yield  identical 
estimates  and  confidence  regions.  Comparisons  of  individual  trial  predictions  confirm 
that  the  actual  predictions  were  identical.  However,  for  the  trials  conducted  under  the 
relatively  stable  conditions,  which  was  also  when  the  wind  speed  was  smallest,  the 
setting  of  uu(calm),  not  surprisingly,  had  a  large  impact  on  the  predictions.  This  impact 
is  reflected  in  the  estimates  of  MOE  2  AE1  and  MOE  2  AE2  shown  in  Figure  3-11. 

For  the  stable  trials,  the  HPAC  predictions  that  were  completed  using  the  default 
value  of  uu(calm)  of  0.25,  result  in  substantially  wider  plumes  and,  hence,  decreased  Afn 
and  increased  App  at  least  for  AE1  estimates.  This  result  is  statistically  significant  for 
MOE  2  AE1.  The  HPAC  MOE  2  AE1  result  for  the  stable  trials  with  the  default  setting 
of  uu(calm)  is  much  more  similar  to  the  NARAC  result  described  earlier  (e.g.,  compare 
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Figure  3-11.  MOE  2,  AE1  and  AE2:  uu(calm)  =  0.0  Versus  uu(calm)  =  0.25 
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to  Figure  3-7,  top,  left  chart).  The  MOE  2  estimates  based  on  AE2,  shown  in  Figure  3- 
11,  suggest  decreased  Afn  and  little  change  (or  perhaps  a  slight  decrease)  in  App.  Based 
on  these  results,  one  can  conclude  that  the  default  value,  uu(calm)  =  0.25,  leads  to 
slightly  improved  predictions  for  the  Prairie  Grass  trials  relative  to  the  baseline  value  of 
uu(calm)  =  0.0.  In  this  sense,  at  least  for  the  Prairie  Grass  field  trials,  the  usage  of  the 
default  value  for  uu(calm)  is  validated  relative  to  the  value  of  0.0. 

Similarly,  we  can  consider  the  MOE  1  AE2  results  for  HPAC  predictions  of  the 
more  stable  trials  done  with  the  default  and  baseline  uu(calm)  values.  The  MOE  1  AE2 
values  for  the  default  and  baseline  uu(calm)  predictions  of  the  SCG  =5,6,7  trials  were 
0.456  (95th  percent  confidence  interval  of  0.393  to  0.521)  and  0.403  (95th  percent 
confidence  interval  of  0.311  to  498),  respectively.  The  MOE  1  AE2  values  for  the 
default  and  baseline  uu(calm)  predictions  of  the  SCG  =  6,7  trials  were  0.446  (95th  percent 
confidence  interval  of  0.394  to  0.503)  and  0.335  (95th  percent  confidence  interval  of 
0.228  to  0.455),  respectively.  Examination  of  MOE1  AE1  results  showed  a  similar  trend. 
In  no  case  were  the  95th  percent  confidence  intervals  separated.  However,  these  MOE  1 
results  do  tend  to  confirm  an  improvement  for  the  HPAC  predictions  of  the  Prairie  Grass 
field  trials  that  used  the  default  value  of  uu(calm). 

Figure  3-12  provides  an  example  of  the  differences  between  the  two  HPAC 
predictions  for  a  trial  that  was  conducted  during  stable  conditions.  The  suggestion,  at 
least  for  stable  trials,  is  that  differences  between  HPAC  and  NARAC  predictions 
obtained  when  using  the  default  setting  of  uu(calm)  are  minimized  relative  to  the  baseline 
uu(calm)  value  predictions.  However,  differences  between  HPAC  and  NARAC 
predictions  of  neutral  and  unstable  Prairie  Grass  trials  persist. 

Appendix  G  provides  comparisons  of  these  two  sets  of  HPAC  Prairie  Grass 
predictions  for  all  of  the  trials.  Of  course,  for  the  higher  wind  speed  neutral  and  unstable 
trials,  the  two  sets  of  HPAC  predictions  were  identical. 
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Dark  Blue:  uu(calm)  =  0  Over-Prediction  Relative  to  uu  (calm)  =  0.25 
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Figure  3-12.  HPAC  Predictions  for  Trial  58  with  Two  Values  of  uu(calm): 
SCG  6  (“Stable”)  at  the  5  Arc  Ranges 
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4.  NARAC  Predictions:  Excursions 


This  section  describes  the  results  of  comparisons  of  the  NARAC  predictions  of 
Prairie  Grass  trials  for  two  excursions.  First,  two  sets  of  NARAC  predictions  were 
prepared.  Both  sets  used  an  experimentally  determined  value  for  the  NARAC  input 
sigma-v  -  a  parameter  related  to  the  dispersion.15  One  of  these  runs  included  a  defined 
deposition  velocity  thus  including  in  the  prediction  surface  deposition  of  some  of  the 
SOo.16  Therefore,  these  first  two  sets  of  NARAC  predictions  allow  for  the  comparison  of 
predictions  “with”  and  “without”  surface  deposition. 

Next,  we  compare  two  sets  of  NARAC  predictions  of  the  Prairie  Grass  field  trials 
that  do  not  include  a  mechanism  for  surface  deposition  of  SCE.  One  of  these  sets 
includes  the  experimentally  determined  value  for  the  parameter  sigma-v.  The  second  set 
uses  the  calculated  value  for  sigma-v  and  is  identical  to  the  set  of  predictions  that  were 
used  for  the  comparisons  to  HP  AC  apart  from  the  change  in  wind  fields  for  four  trials  as 
discussed  in  footnote  13. 

a.  With  and  Without  Surface  Deposition 

Including  surface  deposition  should  lead  to  the  removal  of  some  SCH,  although 
perhaps  only  a  small  amount  of  material.  Therefore,  with  respect  to  using  the  summed 
dosages  (AE2)  to  estimate  App,  Afn  and  A0v  for  MOE  2,  we  expect  that  the  decreases  in 
predicted  material  would,  overall,  lead  to  decreases  in  false  positive  and  increases  in  false 
negative  areas  relative  to  the  “no  deposition”  predictions.  This  is  exactly  the  same 
hypothesis  that  we  applied  to  the  HP  AC  surface  deposition  prediction  excursions.  Again, 
we  note  however,  that  if  deposition  is  a  physically  important  effect  for  the  Prairie  Grass 
field  trials  and  it  is  well-modeled,  then  the  “deposition”  results  should  lead  to  reduced 
false  positive  and  reduced  (or  perhaps  only  slightly  increased)  false  negative  fractions 
relative  to  the  “no  deposition”  predictions. 


15  The  input  weather  for  these  two  sets  of  predictions  differed  slightly  from  the  original  baseline  NARAC 
predictions  that  were  compared  to  HP  AC.  For  trials  35,  39,  51,  and  59,  the  rawinsonde  data  were 
included  in  the  surface  wind  field  computation  and  the  non-divergent  adjustment  was  suppressed.  This 
improved  the  plume  direction  for  cases  in  which  there  was  large  vertical  and/or  horizontal  wind  shear, 
but  had  minimal  effect  in  other  cases.  These  input  weather  changes  resulted  from  an  examination  of 
differences  between  NARAC,  HPAC,  and  field  trial  plume  directions  for  individual  trials  and  arcs. 
Figures  E-27a,  E-31a,  E-42a,  and  E-48a  show  the  differences  between  NARAC,  HPAC,  and  field  trial 
plume  directions  for  the  four  original  baseline  trials  (i.e.,  before  the  changes  to  NARAC  wind  field 
computations  described  in  this  footnote).  Additional  details  associated  with  the  NARAC  predictions 
can  be  found  in  Reference  3-2. 

1()  See  Appendix  B  for  additional  details. 
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Figure  3-13  presents  AE2-based  MOE  2  comparisons  for  NARAC  predictions 
with  and  without  the  inclusion  of  a  surface  deposition  mechanism.  In  addition  to  a 
comparison  of  results  with  all  51  trials,  this  figure  also  presents  results  as  a  function  of 
arc  range  and  SCG. 

In  all  cases,  the  inclusion  of  surface  deposition  in  the  NARAC  prediction  (the 
reddish  confidence  regions  in  Figure  3-13)  led  to  slightly  increased  false  negative  and 
slightly  decreased  false  positive  areas.  For  any  individual  comparison  (e.g.,  400-meter 
arc),  the  confidence  regions  for  the  two  MOE  2  estimates  have  considerable  overlap. 
With  respect  to  App  and  Afn,  the  same  differences  are  seen  for  all  trials  and  for  all 
conditions  examined  (all  ranges  and  all  SCGs).  As  was  true  for  the  HP  AC  deposition 
prediction  excursion,  this  consistency  leads  us  to  conclude  that  MOE  2  AE2  is  able  to 
discern  the  difference  in  NARAC  Prairie  Grass  predictions  due  to  the  inclusion  of  a 
surface  deposition  mechanism.17 

The  magnitudes  of  the  MOE  2  differences  between  the  “with  deposition”  and 
“without  deposition”  trials  are  smaller  than  the  uncertainty  associated  with  the  individual 
point  estimates.  An  inspection  of  the  comparative  figures  of  Appendix  L  shows  just  how 
small  the  differences  in  NARAC  “deposition”  and  “no  deposition”  Prairie  Grass 
predictions  are.  These  results  are  consistent  with  those  previously  described  for  the 
HPAC  surface  deposition  prediction  excursions. 

Additional  inspection  of  the  figures  of  Appendix  L,  presented  on  the  logarithmic 
scale  (the  odd-numbered  pages  in  Appendix  L),  show  the  occasional  over-prediction  of 
dosages  by  the  “with  deposition”  predictions  relative  to  the  “without  deposition” 
predictions.  The  level  of  over-prediction  is  quite  small  and  typically  appears  to  occur  on 
the  “edges”  of  the  predicted  plume  (near  the  threshold  cutoff  value).  This  is  most  likely 
due  to  statistical  error  in  the  NARAC  Monte  Carlo  dispersion  simulations  in  these 
regions  caused  by  the  lower  number  of  particles  used  to  calculate  dosages  toward  the 
edges  of  the  plumes. 

At  this  point,  as  we  did  for  the  HPAC  predictions,  we  might  ask  if  the  decrease  in 
false  positive  fraction  is  greater  than  the  increase  in  false  negative  fraction  associated 
with  the  inclusion  of  surface  deposition.  Again,  we  use  MOE  1  AE2  (with  Cfn  =  Cfp  = 
1.0)  to  aid  this  assessment 


17  Other  measures,  including  scatterplots  of  observed  and  predicted  values,  showed  similar  trends.  [Ref. 
3-2.] 
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Figure  3-13.  MOE  2,  AE2:  NARAC  Predictions  With  and  Without  Surface  Deposition 


Table  3-4  compares  MOE  1  AE1  values  for  NARAC  predictions  done  with  and 
without  surface  deposition.  The  numbers  in  parentheses  correspond  to  the  95th  percent 
confidence  intervals  associated  with  the  given  point  estimate.  The  larger  MOE  1  AE2 
value  for  each  comparison  is  highlighted  in  blue  boldfaced  text. 
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Table  3-4.  MOE  1  AE2  Comparisons  for  NARAC  Predictions  of  Prairie  Grass  Field  Trials: 

With  and  Without  Surface  Deposition 


Trial  Type 

With 

Without 

All 

0.586  (0.551  -0.616) 

0.584  (0.547-0.616) 

Range  =  50  m 

Range  =  100  m 

Range  =  200  m 

Range  =  400  m 

Range  =  800  m 

0.629  (0.593-0.663) 

0.600  (0.563  -  0.634) 

0.573  (0.529-0.614) 

0.501  (0.455-0.544) 

0.351  (0.302-0.398) 

0.623  (0.586-0.654) 

0.593  (0.557-0.627) 

0.570  (0.531  -0.611) 

0.509  (0.465  -  0.554) 

0.363  (0.314-0.412) 

SCG  =  1,2 

SCG  =  1,2,3 

SCG  =  3,4,5 

SCG  =4 

SCG  =  5,6,7 

SCG  =  6,7 

0.632  (0.589-0.673) 

0.655  (0.619-0.691) 

0.598  (0.554-0.641) 

0.587  (0.551  -0.622) 

0.463  (0.404-0.522) 

0.446  (0.354-0.525) 

0.624  (0.582-0.667) 

0.649  (0.611  -0.682) 

0.596  (0.549-0.640) 

0.587  (0.547-0.624) 

0.470  (0.406  -  0.529) 

0.460  (0.356-0.541) 

For  most  conditions  shown  in  the  Table  3-4  the  “with  deposition”  predictions  led 
to  a  marginally  improved  MOE  1  AE2  value  (indicating  associated  larger  decreases  in 
false  positive  fraction  than  the  corresponding  increases  in  false  negative  fraction).  The 
exceptions  to  the  above  statement  are  the  results  for  the  400-meter  arc,  the  800-meter  arc 
and  for  the  more  stable  trials.  In  fact,  for  the  SCG  =  5,6,7  and  SCG  =  6,7  the  “without 
deposition”  MOE  1  AE2  value  is  slightly  larger  than  the  “with  deposition”  result. 

For  all  of  the  differences  shown  in  Table  3-4,  the  95th  percent  confidence  intervals 
overlap  (as  can  be  seen  in  the  table).  We  also  inspected  the  80th  percent  confidence 
intervals  and  found  the  same  result  (i.e.,  in  all  cases  the  intervals  overlap).  Thus,  the 
differences  between  individual  comparisons  shown  in  Table  3-4  are  not  considered 
statistically  significant.  Therefore,  although  the  trend  suggested  by  MOE  1  AE2  is 
toward  improvement  with  the  inclusion  of  a  surface  deposition  mechanism,  at  least  at  the 
shorter  ranges  and  with  the  exception  of  the  more  stable  trails,  we  consider  these  results 
marginal  at  best. 

As  was  the  case  with  the  HPAC  “with  and  without  deposition”  prediction 
comparisons  the  cause  for  the  stable  trials  exception  described  above  can  be  discerned 
from  a  look  at  some  of  the  individual  trial  results.  Figures  L-23a  (trial  28,  SCG  =  5), 
L-24a  (trial  32,  SCG  =  6),  and  L-48a  (trial  59,  SCG  =  5), 18  suggest  the  same  possible 


l8  Page  L-l  describes  the  meanings  of  the  various  shadings  in  the  Appendix  L  figures. 
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cause  for  this  result  as  was  hypothesized  for  the  HPAC  comparisons.  In  each  of  these 
cases,  the  baseline  NARAC  prediction  (“without  deposition”)  led  to  an  under-prediction 
of  a  very  narrow  peak.  The  “with  deposition”  predictions  simply  removed  material  from 
the  predicted  peak  (particularly  at  the  400-  and  800-meter  arc),  therefore  leading  to  a 
larger  AE2-based  false  negative  fraction  or  area. 

In  conclusion,  although  differences  were  detected,  the  addition  of  a  deposition 
mechanism  to  the  modeling  did  not  lead  to  a  clear  improvement  in  the  MOE  2  value  for 
the  HPAC  predictions.  Furthermore,  inspection  of  MOE  1  AE2  values,  with  the 
assumption  of  equal  weighting  of  false  positive  and  false  negative  fractions,  suggested,  at 
best,  marginal  overall  (MOE  1  AE1  (All  Trials)  =  0.586  versus  0.584)  improvement  with 
the  inclusion  of  a  surface  deposition  mechanism. 

b.  With  Different  Dispersion  Parameter  Values:  Sigma-v  Experimental  and 

Sigma-v  Calculated 

Figure  3-14  compares  NARAC  predictions  of  Prairie  Grass  as  a  function  of  SCG. 
MOE  2  AE2  results  are  shown  for  predictions  using  experimentally  observed  sigma-v 
values  input  to  the  NARAC  modeling  system  and  for  sigma-v  calculated  by  the  NARAC 
modeling  system.  For  the  neutral/near  neutral  trials  (SCG  =  4  and  3,4,5)  the  confidence 
regions  for  the  MOE  2  AE2  point  estimate  overlap  almost  entirely.  That  is,  there  does 
not  appear  to  be  any  difference.  For  both  the  stable  and  unstable  trials,  the  predictions 
using  the  experimentally  observed  sigma-v  appear  to  be  a  small  improvement,  moving 
toward  decreased  false  negative  and  decreased  false  positive  fractions.  These 
improvements  in  MOE  2  AE2  appear  small,  on  the  order  of  the  uncertainty  in  the 
estimate.  For  example,  the  larger  95th  percent  confidence  regions  associated  with  the 
NARAC  “Sigma-v  Exp”  predictions  of  the  stable  trials  (SCG  =  6,7  and  5,6,7)  completely 
contain  the  “Sigma-v  Calc”  predictions.  It  is  not  surprising  that  the  use  of  the 
experimentally  measured  sigma-v  value  improves  the  NARAC  model  predictions.  The 
relatively  close  agreement  between  the  NARAC  results  with  and  without  the 
experimentally  observed  sigma-v  suggests  that  the  NARAC  parameterizations  used  to 
calculate  sigma-v  are  reasonable. 

Table  3-5  compares  MOE  1  AE2  estimates  for  NARAC  predictions  done  with  the 
experimental  and  calculated  sigma-v  values.  These  values  are  for  Cfn  =  CFp  =1.0  (i.e., 
equal  weighting  of  Afn,  Afp,  and  Aov).  The  numbers  in  parentheses  correspond  to  the 
95th  percent  confidence  intervals  associated  with  the  given  point  estimate.  The  larger 
MOE  1  AE2  value  for  each  comparison  is  highlighted  in  blue  boldfaced  text. 


3-27 


By  SCG 


m 

k 

95th%  Confiden 

ce  Region 

s 

0  0.2  0.4  0.6  0.8  1 


Decreasing  False  Negative 

O  Sigma-v  Exp,  SCG  =  1 ,2,3  ♦  Sigma-v  Exp,  SCG  =  4 

X  Sigma-v  Exp,  SCG  =  5,6,7  o  Sigma-v  Calc,  SCG  =  1 ,2,3 

♦  Sigma-v  Calc,  SCG  =  4  Sigma-v  Calc,  SCG  =  5,6,7 


By  SCG 


Q-  0.6 


0.4 


4 

► 

95th%  Confidence  Regions 

0  0.2  0.4  0.6  0.8 


Decreasing  False  Negative 

o  Sigma-v  Exp,  SCG  =  1,2  ♦  Sigma-v  Exp,  SCG  =  3,4,5 

Sigma-v  Exp,  SCG  =  6,7  o  Sigma-v  Calc,  SCG  =  1 ,2 

♦  Sigma-v  Calc,  SCG  =  3,4,5  Sigma-v  Calc,  SCG  =  6,7 


Figure  3-14.  MOE  2,  AE2:  NARAC  Predictions  Using  Calculated  and  Experimental  Sigma-v 


Table  3-5.  MOE  1  AE2  Comparisons  for  NARAC  Predictions  of  Prairie  Grass  Field  Trials: 

Calculated  and  Experimental  Sigma  -v 


Trial  Type 

Experimental 

Calculated 

All 

0.586  (0.553-0.619) 

0.569  (0.530-0.604) 

Range  =  50  m 

Range  =  100  m 

Range  =  200  m 

Range  =  400  m 

Range  =  800  m 

0.629  (0.592-0.661) 

0.600  (0.563-0.635) 

0.573  (0.527-0.619) 

0.501  (0.457-0.544) 

0.351  (0.303-0.395) 

0.615  (0.576-0.648) 

0.585  (0.546-0.623) 

0.550  (0.505-0.592) 

0.471  (0.429-0.517) 

0.336  (0.285-0.378) 

SCG  =  1,2 

SCG  =  1,2,3 

SCG  =  3,4,5 

SCG  =4 

SCG  =  5,6,7 

SCG  =  6,7 

0.632  (0.588-0.671) 

0.655  (0.619-0.691) 

0.598  (0.549  -  0.642) 

0.587  (0.552-0.621) 

0.463  (0.402-0.517) 

0.446  (0.353  -  0.527) 

0.604  (0.545-0.658) 

0.628  (0.583-0.669) 

0.596  (0.547-0.636) 

0.603  (0.551  -  0.652) 

0.445  (0.396-  0.504) 

0.413  (0.355-0.469) 

For  most  of  the  trial  types  considered  in  Table  3-5,  the  predictions  that  used  the 
experimental  sigma-v  led  to  marginally  improved  MOE  1  AE2  values.  The  SCG  =  4 
trials’  predictions,  based  on  a  sample  size  of  10,  represent  the  exception.  In  all  cases, 
comparisons  led  to  overlap  between  95th  (and  80th)  percent  confidence  intervals.  That  is, 
no  individual  comparison  appears  to  be  statistically  significant.  Appendix  M  provides 
individual  trial  comparisons  for  the  sigma-v  experimental  and  sigma-v  calculated 
NARAC  predictions. 
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B.  A  FEW  COMMENTS  ON  THE  “STANDARD”  STATISTICAL  MEASURES 


In  addition  to  the  MOEs  described  in  the  previous  sections,  we  computed  the 
values  of  several  standard  statistical  measures.  In  large  part,  the  goal  of  these 
computations  was  to  allow  for  comparisons  of  conclusions  based  on  MOE  values  to 
conclusions  based  on  standard  statistical  measure  examinations.  These  measures,  which 
include  FB,  NMSE,  MG,  VG,  R,  FAC2,  FAC5,  and  FAC10,  are  described  in  Chapter  2, 
Section  A. 2.  Appendix  I  provides  detailed  results  (via  plots  and  tables)  associated  with 
these  measures. 

Figure  3-15  provides  comparisons  (to  field  trial  observations)  of  HPAC  and 
NARAC  predicted  ArcMax  for  the  measures  FB,  NMSE,  MG,  and  VG.  The  display  on 
the  left  presents  point  estimates  and  95th  percent  confidence  intervals  for  FB  and  NMSE. 
The  display  on  the  right  presents  point  estimates  and  95th  percent  confidence  intervals  for 
MG  and  VG.  The  blue  parabolas  correspond  to  limiting  curves  -  since  NMSE  and  VG 
for  a  model  can  be  due  to  mean  bias  and  random  error,  there  are  minimum  values  of 
NMSE  and  VG  for  a  given  FB  and  MG,  respectively.19 


FB\&NVEE 


VGvsMS 


Figure  3-15.  FB,  NMSE,  MG,  and  VG:  HPAC  and  NARAC  ArcMax  Comparisons 

to  Prairie  Grass  Trials 

The  x-axis  for  the  plots  above  describes  the  overall  bias  with  respect  to  a  model’s 
ability  to  predict  the  ArcMax.  The  vertical  lines  at  MG  =  1  and  -1  in  the  VG  versus  MG 
plot  coincide  with  a  result  that  indicates  a  factor-of-two  difference  between  the  prediction 


19  See  Appendix  I  for  a  brief  discussion. 
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and  the  observation.  FB  values  greater  than  0.0  or  MG  values  greater  than  1.0  imply,  on 
average,  an  under-prediction  by  the  model.  Similarly,  FB  values  less  than  0.0  or  MG 
values  less  than  1.0  suggest  an  over-prediction,  on  average,  by  the  model. 

Both  plots  imply  that  NARAC  tends  to  underpredict  the  peak  (ArcMax)  - 
NARAC  FB  =  0.26  (with  a  95th  percent  lower  confidence  limit  of  0.12)and  NARAC  MG 
=  1.18  (with  a  95th  percent  lower  confidence  limit  of  1.11).  For  HPAC,  the  FB  value  of 
-0.30  (with  a  95th  percent  upper  confidence  limit  of  -0.12)  implies  that  the  HPAC 
predictions,  on  average,  overpredict  the  ArcMax.  The  HPAC  MG  value,  1.04  (with  a  95th 
percent  confidence  interval  of  0.97  to  1.11)  is  not  significantly  different  from  1.0.  The 
confidence  intervals  for  HPAC  and  NARAC  MG  and  FB  estimates  are  completely 
separated,  indicating  that  predictions  of  the  two  modeling  systems  are  significantly 
different  from  each  other. 

With  respect  to  NMSE  and  VG,  both  measures  of  variance,  there  is  no  statistical 
difference  (see  the  95th  percent  intervals)  between  the  two  models’  predictions.  The 
larger  confidence  interval  associated  with  the  HPAC  NMSE  estimate  is  due  to  just  a  few 
large  values.20 

Figure  3-16  provides  comparisons  (to  field  trial  observations)  of  HPAC  and 
NARAC  predicted  CWI  (crosswind  integrated)  dosage  for  the  measures  FB,  NMSE,  MG, 
and  VG.  These  plots  indicate  that  HPAC  tends  to  underpredict  CWI  dosage  (HPAC  FB 
=  0.16  [0.00  to  0.26]21  and  HPAC  MG  =  1.20  [1.14  to  1.25]).  Based  on  MG,  NARAC 
tends  to  overpredict  CWI  dosage  (NARAC  MG  =  0.82  [0.79  to  0.85]).  The  NARAC  FB 
value  of  -0.06  [-0.18  to  0.02]  is  not  significantly  different  from  0.00.  The  MG  results  for 
HPAC  and  NARAC  are  significantly  different. 

With  respect  to  VG(CWI  dosage)  the  results  for  the  two  models  are  similar  given 
the  uncertainty  estimates. 

In  addition  to  the  above  standard  measures  that  dealt  with  ArcMax  and  CWI 
dosage,  we  examined  an  estimate  of  the  plume’s  width,  Wec,  as  described  in  Chapter  2. 
Figure  3-17  presents  the  FB  values  as  a  function  of  range  for  Wec.  The  triangles  and 
diamonds  correspond  to  the  point  estimates  for  FB(Wec),  the  solid  red  lines  correspond  to 


20  This  sort  of  limitation  for  this  measure,  NMSE,  was  described  in  Chapter  2.  Most  of  the  variance  in 
NMSE  is  associated  with  a  few  stable  trials,  particularly  the  50-  and  800-meter  arcs.  See  Figures  1-1 
and  1-3. 

- '  Numbers  in  brackets  correspond  to  the  95th  percent  confidence  interval. 
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the  80th  percent  confidence  intervals,  and  the  solid  blue  lines  correspond  to  the  95th 
percent  confidence  intervals.  The  negative  NARAC  FB  values  imply  that  the  observed 
plume  was,  on  average,  narrower  than  the  NARAC-predicted  plume.  Similarly,  the 
positive  FB  values  associated  with  the  HP  AC  calculations  imply  that  the  observed  plume 
was,  on  average,  wider  than  the  HP  AC  predictions.  For  three  of  the  arc  ranges,  200,  400, 
and  800  meters,  the  NARAC  Wec  values  include  the  perfect  FB  value  of  0.0.  The 
NARAC-predicted  Wec  values  are  in  better  agreement  with  the  observations  than  the 
baseline  HP  AC-predicted  values. 


FB\&  NVEE 


Figure  3-16.  FB,  NMSE,  MG,  and  VG:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to 

Prairie  Grass  Trials 
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Figure  3-17.  FB  as  a  Function  of  Range:  HPAC  and  NARAC  Wec 
Comparisons  to  Prairie  Grass  Trials 
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Figures  1-16,  1-17,  and  1-18  (Appendix  I)  provide  additional  comparisons  of 
HP  AC  and  NARAC  FB  (Wec)  and  NMSE  (Wec)  as  a  function  of  range  and  stability 
category.  Figure  1-17  suggests  that  HP  AC  under-predictions  of  Wec  are  due  mainly  to  the 
stable  trials  (SCG  =  6,7).  However,  HP  AC  predictions  done  with  the  default  uu(clam) 
value  of  0.25  lead  to  a  substantial  reduction  in  the  under-prediction  of  Wec  for  the  stable 
trials.22 

The  above  results  suggest  that,  on  average  and  as  the  models  were  run,  HP  AC 
produces  narrower  and  sharper  plumes  relative  to  NARAC’s  wider  plumes.  Figure  3-18 
provides  two  examples  that  tend  to  confirm  this  explanation.  A  more  complete 
inspection  of  the  figures  of  Appendix  E  supports  this  contention.  This  conclusion  is 
consistent  with  our  findings  based  on  examinations  of  MOEs  -  the  NARAC  predictions 
led  to  larger  false  positive  fractions  (i.e.,  wider  plumes  at  a  given  threshold)  and  the 
HPAC  predictions  led  to  larger  false  negative  areas  (i.e.,  narrower  plumes). 

With  respect  to  comparisons  of  the  standard  statistical  measures  and  the  MOEs, 
we  have  the  following  comments: 

.  For  these  data,  the  conclusions  based  on  the  MOEs  and  the  standard  statistical 
measures  were  consistent. 

.  Individual  standard  statistical  measures  were,  in  general,  less  likely  to  detect 
statistically  significant  differences  (e.g.,  as  a  function  of  arc  range)  relative  to 
the  MOEs.  This  may  be  due  to  the  fact  that  the  MOEs  are  based  on  point-to- 
point  comparisons  and  include  directional  effects.  The  standard  statistical 
measures  that  we  examined  did  not  include  these  effects. 

Perhaps  the  most  important  difference  between  the  MOEs  and  standard  measures 
included  in  this  paper  is  related  to  information  content  of  the  parameters  that  were 
examined.  The  statistical  measures  that  we  used  were  applied  to  ArcMax,  CWI  dosage, 
and  Wec.  That  is,  each  arc  was  characterized  by  a  peak,  a  total  dosage,  and  a  width. 
Information  associated  with  the  relative  direction  of  the  observed  and  predicted  plumes 
was  not  included  in  the  standard  measure  analysis.  For  some  applications,  assessing 
model  performance  in  this  way  may  be  perfectly  acceptable,  for  instance,  if  one  is  only 
interested  in  the  dispersion  features  of  the  model.  However,  if  transport  (e.g.,  direction) 
and  dispersion  are  of  interest,  then  the  measures  cited  above  could  not  be  used. 


22  See  Figures  G-8,  G-9,  G-24,  G-27,  G-28,  G-31,  G-32,  and  G-47  (SCG  =  6.  7  trials). 
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Figure  3-18.  HPAC  and  NARAC  Predictions  for  Prairie  Grass  Trials  28  and  45 


As  described  earlier,  the  MOEs  directly  allow  for  assessments  that  include 
differences  in  predicted  and  observed  directions.  Of  course,  for  such  measures,  it  will  be 
important  to  sort  out  the  causes  of  substantial  differences  between  predictions  and 
observations.  Whereas  an  incorrect  wind  direction  measurement  would  have  no  impact 
on  measures  that  considered  ArcMax  or  CWI  dosage,  such  an  error  could  dramatically 
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change  the  result  of  an  MOE  1  or  MOE  2  calculation.  Additional  research  could  be  done 
to  compare  the  MOEs  against  statistical  measures  that  incorporate  directional  effects. 

To  further  explore  the  above,  we  examined  CentDir  -  that  is,  the  centerline 
direction  (in  degrees)  as  described  in  Chapter  2  and  Appendix  J.  The  calculated  bias  and 
RMSE  values  associated  with  the  CentDir  parameter  are  deposited  in  Appendix  I.  Figure 
3-19  presents  an  interesting  result.  First,  the  precision  by  which  the  CentDir  parameter 
was  measured  is  at  best  equal  to  the  angular  separation  between  samplers  (i.e.,  1  degree 
for  the  800-meter  arc  and  2  degrees  for  the  rest  of  the  arcs).  The  point  estimates  and 
confidence  intervals  shown  in  Figure  3-19  suggest  that  the  observations  and  predictions 
(both  HPAC  and  NARAC)  have  about  the  same  centerline  direction,  within  the  precision 
of  this  measurement,  out  to  about  400  meters.  Interestingly,  the  800-meter  bias  values 
(and  the  overall  trend  shown)  suggest  that  there  may  be  a  systematic  error  associated  with 
the  Prairie  Grass  field  trial  wind  direction  measurements. 
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Figure  3-19.  Bias  as  a  Function  of  Range:  HPAC  and  NARAC  CentDir 
Comparisons  to  Prairie  Grass  Trials 
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C.  CONCLUSIONS:  VALUE  OF  MOEs 


In  this  chapter,  novel  user-oriented  MOEs  have  been  used  to  compare  Prairie 
Grass  field  trial  observations  to  HPAC  and  NARAC  predictions.  These  MOEs  provide  a 
useful  tool  for  examining  differences  in  model  performance.  This  application  has 
demonstrated  their  relative  ease  of  use,  at  least  for  a  short-range  field  trial.  These 
measures  can  communicate  numerically  what  plume  plots  show  graphically. 

Our  user-oriented  MOEs  are  defined  in  terms  of  false  positive,  false  negative,  and 
overlap  regions.  The  choice  of  area  estimate  allows  the  MOEs  to  be  used  to  compare 
different  quantities,  for  example,  the  degree  of  under-/over-prediction  for  a  critical 
threshold  area  (AE1)  or  for  spatially  integrated  dosages  (AE2).  The  MOEs  can  also  be 
interpreted  by  first  applying  an  effects  model,  e.g.,  a  probit  curve.  This  methodology 
allows  a  user  to  evaluate  a  model’s  performance  directly  in  terms  of  an  exposed  or 
potentially  exposed  population. 

The  two-dimensional  measure  (MOE  2),  with  false  positive  and  false  negative 
fractions  considered  orthogonal  consistently  appears  to  resolve  important  features.  MOE 
2  was  useful  in  identifying  and  describing  differences  between  HPAC  and  NARAC 
predictions  of  the  Prairie  Grass  trials.  Variances,  in  a  given  model’s  prediction 
performance  as  a  function  of  arc  range  and  stability  condition,  were  easily  detected  and, 
often,  led  to  statistically  significant  conclusions.  Examination  of  MOE  2  values  and 
trends  allowed  us  to  discern  prediction  performance  differences,  based  on  changes  in 
HPAC  inputs  (i.e.,  including  a  mechanism  for  SO2  deposition  and,  separately,  modifying 
the  uu(calm)  default  value).  Similarly,  input  changes  to  NARAC  predictions  (i.e., 
including  a  mechanism  for  SO2  deposition  and,  separately,  modifying  the  sigma-v  value) 
led  to  detectable  differences  in  prediction  performance  (i.e.,  movement  in  MOE  2  space). 
Furthermore,  we  found  that  the  2D  MOE  (MOE  2)  allowed  for  straightforward 
communication  of  the  impact  of  these  modest  input  changes  in  terms  of  false  positive  and 
false  negative  fractions. 

We  found  that  a  lethality/effects  filter  can  be  used  to  compute  MOE  values,  and 
can  be  characterized  as  a  “tunable  dial”  that  can  relate  the  goodness  of  a  prediction  for 
agents  of  greatly  varying  toxicity  (e.g.,  biological  versus  chemical  agents  or  threshold 
versus  lethal  effects  for  a  chemical  agent).  This  feature  may  make  this  methodology  of 
particular  value  with  respect  to  user  accreditation.  For  instance,  the  specific  application 
and  user  will  dictate  the  agent  type,  effect,  and  downwind  distance  of  interest,  therefore 
defining  how  the  lethality/effects  dial  should  be  tuned. 
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Overall  the  2D  MOE  revealed  the  following  with  respect  to  comparisons  of 
HP  AC  and  NARAC  predictions  under  the  specified  model  intercomparison  protocol: 

.  HP  AC  and  NARAC  predictions  of  the  relatively  unstable  Prairie  Grass  trials 
were  most  similar.  Predictions  of  the  most  stable  trials  led  to  the  biggest 
differences. 

.  In  a  sense,  the  NARAC  predictions,  as  run,  were  more  conservative  than  the 
HPAC  predictions.  In  general,  NARAC  predictions  led  to  substantially 
smaller  false  negative  areas,  but  at  the  expense  of  larger  false  positive  areas. 
This  was  most  true  for  the  predictions  of  the  neutral  and  stable  trials.  In 
contrast,  HPAC  predictions  showed  substantially  smaller  false  positive  areas, 
but  at  the  expense  of  larger  false  negative  areas.  A  review  of  Appendix  E 
confirms  this  interpretation. 

This  study  also  used  a  variety  of  other  statistical  and  graphical  measures  of  model 
performance.  Examination  of  plots  of  measured  and  predicted  dosages  as  a  function  of 
distance  helped  support  conclusions.  Statistical  measures  for  derived  quantities  -  such  as 
ArcMax,  CWI  dosage,  plume  horizontal  spread,  and  mean  plume  direction  -  were  of 
some  value  in  confirming  the  underlying  reasons  for  the  MOE  results.  MOE  results  (for 
both  AE1  and  AE2)  include  the  effects  of  directional  error  by  pairing  observed  and 
predicted  dosage  in  space  (and  time).  Separate  comparisons  of  ArcMax,  CWI  dosage, 
plume  spread,  and  mean  plume  direction  (which  do  not  measure  mean  wind  direction 
error)  can  allow  for  the  isolation  of  errors  due  to  the  dispersion  modeling  methods  from 
those  due  to  mean  wind  advection  modeling  errors  (particularly  directional).  The  results 
of  this  study  show  that  MOEs  are  a  potentially  valuable  tool,  but  that  no  single  measure 
can  be  used  exclusively  in  evaluating  and  analyzing  model  performance. 

Additional  work  can  be  pursued  to  further  develop,  improve,  and  demonstrate  the 
value  of  the  new  MOEs.  It  might  be  valuable  to  compare  the  new  MOEs  to  standard 
statistical  measures  that  are  applied  to  dosage  observations  and  predictions  that  are  paired 
in  space  and  time,  since  this  is  the  approach  used  when  determining  the  area  estimates  for 
the  new  MOEs.  Similarly,  since  the  MOEs  differentiate  over-predictions  from  under¬ 
predictions,  this  might  also  be  done  for  the  standard  statistical  measures  in  order  to 
determine  the  unique  features  of  the  MOEs.  In  particular,  it  would  be  of  interest  to 
understand  further  how  much  of  the  differences  identified  by  the  MOEs  are  due  to  plume 
directional  errors. 

Further  study  of  the  sensitivity  of  MOE  AE1  to  the  choice  of  critical  threshold 
dosage  could  be  of  some  value.  In  particular,  the  AE1 -based  MOEs,  to  the  degree  that 
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they  allow  the  user  to  set  the  threshold  of  interest,  may  best  be  characterized  as 
corresponding  to  operational  or  accreditation  measures.23 

At  this  point,  it  appears  possible  to  extend,  or  possibly  adapt,  the  new  MOEs,  to 
evaluations  that  consider  longer-range  field  trial  observations.  These  extensions  and/or 
adaptations  are  to  be  explored  in  the  near  future.  Additional  study,  to  compare  and 
contrast  differing  interpolation  schemes  to  estimate  actual  area  sizes  from  field  trial  data, 
is  underway. 

Future  efforts  will  also  focus  on  developing  a  quantitative  framework  from  which 
a  user  can  describe  his/her  risk  tolerances.  In  this  way,  the  2-dimensional  space 
associated  with  MOE  2  can  be  divided  into  acceptable  and  unacceptable  regions  for  a 
given  user. 


23  The  AE1  methodology,  as  described  previously,  is  identical  to  the  application  of  a  lethality/effects 
filter  in  which  the  lethality/effects  model  is  simply  a  step  function  fi.e.,  a  critical  threshold).  The 
threshold  value  of  60  mg-sec/m3  that  was  used  throughout  this  report  is  higher  than  a  typical  threshold 
effects  dosage  for  the  occurrence  of  ocular  effects  for  nerve  agents. 
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ACRONYMS 


2D 

Two-dimensional 

Aob 

Area  Associated  With  the  Observations 

ADAPT 

Atmospheric  Data  Assimilation  and  Parameterization  Techniques 

AE1 

Area  Estimate  1 

AE2 

Area  Estimate  2 

AFp 

False  Positive  Area 

Afn 

False  Negative  Area 

A&M 

Texas  A&M  University 

Aov 

Area  of  Overlap 

Apr 

Area  Associated  With  the  Prediction 

ARAC 

Atmospheric  Release  Advisory  Center 

ARAP 

Aeronautical  Research  Associates  of  Princeton 

ArcMax 

Maximum  dosage  along  an  arc 

At 

Total  Area 

BCa 

Bias-corrected  and  accelerated 

cal 

calories 

CB 

Chemical  and  Biological 

CBIAC 

Chemical  and  Biological  Defense  Information  Analysis  Center 

cc 

Fractional  Cloud  Cover 

CentDir 

Centerline  Direction 

Cfn 

false  negative  coefficient 

Cpp 

false  positive  coefficient 

cm 

centimeters 

C0 

predicted  dosages/concentrations  or  for  this  study 
dosages/concentrations  predicted  by  HPAC 

COM 

Center  of  Mass 

Cp 

observed  dosages/concentrations  or  for  this  study 

dosages/concentrations  predicted  by  NARAC 

CWI 

Crosswind  Integrated 

DOD 

Department  of  Defense 

DOE 

Department  of  Energy 

DPG 

Dugway  Proving  Ground 

DTRA 

Defense  Threat  Reduction  Agency 

A-l 


ETEX 

FAC  2 
FAC  5 
FAC  10 
FB 
FOM 

GMU 

h 

H0 

H202 

h2so4 

HP  AC 
hr 

IDA 

IDF 

kg 

F 

FCF 
LD50 
FE  (d) 

FFNF 

FODI 

F 

m 

MFT 

MG 

mg 

min 

MIT 

MMD 

MMO 

MOE 

MOF 

MOM 

m/s 

N 

NARAC 


European  Tracer  Experiment 

fraction  of  predictions  within  a  factor  of  2 
fraction  of  predictions  within  a  factor  of  5 
fraction  of  predictions  within  a  factor  of  10 
Fractional  Bias 
Figure  of  Merit 

George  Mason  University 

boundary  layer  height 
Sensible  Heat  Flux 
Hydrogen  Peroxide 
Sulfuric  Acid 

Hazard  Prediction  and  Assessment  Capability 
hour 

Institute  for  Defense  Analyses 
Interactive  Data  Fanguage 

kilogram 

Monin-Obukhov  length  scale 

Fower  Confidence  Fimit 

Fethal  Dosage  (for  50  percent  of  the  population) 

Fethal  Exposure  fraction  -  for  a  dosage  ,d,  the  fraction  of  people 
that  die 

Fawrence  Fivermore  National  Faboratory 
Fagrangian  Operational  Dispersion  Integrator 

microns 

meters 

Model-to-Field  Trial 
geometric  mean  bias 
milligrams 
minutes 

Massachusetts  Institute  of  Technology 

Mass  Median  Diameter 

Model-to-Model-Only 

Measure  of  Effectiveness 

Monin-Obukhov  Fength 

Median  of  Mass 

meters  per  second 

Neutral 

National  Atmospheric  Release  Advisory  Center  (same  as  ARAC) 
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NMSE 
NO  A  A 


R 

rc 

RMSE 

ROC 

a 

S 

SCG 

SCIPUFF 

sec 

S02 


T&D 

TWR 

U 

u* 

UCL 

VG 

V&V 

VV&A 

W 

Wec 

WMD 

ZI 

Zi 

Zo 

Zr 


Normalized  Mean  Square  Error 

National  Oceanic  and  Atmospheric  Administration 


correlation  coefficient 
canopy  resistance 
Root  Mean  Square  Error 
Receiver  Operating  Characteristic 

sigma 

Stable 

Stability  Category  Grouping 
Second-Order  Closure  Integrated  Puff 
seconds 
Sulfur  Dioxide 

Air  Temperature 
Conditional  Averaging 
Transport  and  Dispersion 
Tower 

Unstable 
Friction  Velocity 
Upper  Confidence  Limit 

geometric  mean  variance 

Verification  and  Validation 

Verification,  Validation,  and  Accreditation 

University  of  Wisconsin 

Width  as  estimated  by  the  equal-cuts-off-the-sides  method 
Weapons  of  Mass  Destruction 

boundary  layer  height 

boundary  layer  height 

surface  roughness  height 

reference  height  for  the  deposition  velocity 
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INITIAL  CONDITIONS  FOR  HPAC  AND  NARAC  PREDICTIONS 


This  appendix  describes  the  initial  input  parameters  that  were  used  for  HPAC  and 
NARAC  predictions  of  the  Prairie  Grass  field  trials  [Ref.  B-l  and  B-2]. 

1.  PRAIRIE  GRASS  COMPARISON  PROTOCOL 

This  section  of  the  appendix  is  extracted  from  draft  documentation  [Ref.  B-3]. 
The  preparation  of  initial  input  parameters  for  the  simulations  consisted  of  two  parts. 
First,  meteorological  (e.g.,  wind  speed,  wind  direction,  and  temperature  at  various 
observation  stations)  and  sulfur  dioxide  (SO2)  release  information  was  manually 
transcribed  from  the  hardcopy  of  the  Prairie  Grass  field  trial  report  into  computer-ready 
files.  Second,  data  from  various  tables  in  the  hardcopy  of  the  report  was  used  to  derive 
(or  fit)  additional  input  parameters  as  required  for  each  of  the  simulation  models. 

A  typical  meteorological  input  data  file  for  a  particular  field  trial  is  shown  in 
Figure  B-l.  The  first  line  description  of  the  trial  includes  the  experiment  number 
followed  by  the  fractional  cloud  cover  (in  tenths),1  air  temperature  at  an  altitude  of  2 
meters  (in  degrees  Celsius),2  net  radiation  (in  cal/cm2-sec),3  sensible  heat  flux  (in  /cm2- 
sec),4  latent  heat  flux  (in  /cm2-sec),5  and  friction  velocity  (in  meters/second).6’7 


1  See  Reference  B-l,  Volume  I,  Table  3.1. 

2  See  Reference  B-l,  Volume  II,  Table  8.1. 

3  See  Reference  B-l,  Volume  II,  Table  8.1. 

4  See  Reference  B-l,  Volume  II,  Tables  9.2  and  10.1. 

3  See  Reference  B-l,  Volume  II,  Tables  9.2  and  10.1.  In  the  first  line,  the  “W”  between  the  latent  heat 
flux  and  the  friction  velocity  values  stands  for  the  origin  of  the  sensible/latent  heat  flux  observations. 
If  this  character  is  blank,  then  the  sensible/latent  heat  flux  is  taken  from  Reference  B-l,  Volume  2, 
Table  9.2  (Texas  A&M  observations),  and  if  this  character  is  W,  the  sensible/latent  heat  flux  is  taken 
from  Reference  B-l,  Volume  2,  Table  10.1  (University  of  Wisconsin  observations). 

6  The  friction  velocity  was  calculated  from  the  u'w'  and  v'w'  data  in  Reference  B-2,  Table  17.3. 

7  For  some  trials,  an  additional  data  field  between  the  air  temperature  and  the  net  radiation  fields  is 

populated  with  a  value.  In  these  cases,  this  value  corresponds  to  the  insolation  (in  cal/cm2-sec).  See 
Reference  B-l,  Volume  II,  Table  8.1. 
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Experiment  24  i 

0  21.89 

-0 . 00115 

0 . 03 

0.02  W 

0.404 

560730045000 

RAWINSON 

2 . 00 

3.80 

130.00 

-99999 . 00 

560730045000 

RAWINSON 

290 . 00 

13.30 

150 . 00 

-99999 . 00 

560730045000 

RAWINSON 

600.00 

14 .40 

160.00 

-99999 . 00 

560730045000 

RAWINSON 

930.00 

13 . 70 

190 . 00 

-99999 . 00 

560730045000 

RAWINSON 

1270 . 00 

14.20 

210 . 00 

-99999 . 00 

560730051000 

SOURCE 

2 . 00 

6 . 22 

141 . 00 

7 . 10 

560730051000 

NORTH4  5  0 

2 . 00 

5 . 76 

142 . 00 

6.20 

560730051500 

A&M 

TWR 

0 .25 

3 . 82 

-99999 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

0 . 50 

4 . 61 

-99999 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

1 . 00 

5 .21 

146 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

2 . 00 

5 . 86 

-99999 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

4 . 00 

6 . 62 

-99999 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

8 . 00 

7 . 64 

-99999 . 00 

-99999 . 00 

560730051500 

A&M 

TWR 

16 . 00 

8 . 65 

-99999 . 00 

-99999 . 00 

560730051500 

MIT 

3L 

2.00  - 

99999 . 00 

135.40 

6.40 

Figure  B-1.  Meteorological  Input  File  for  Prairie  Grass  Field  Trial  24 


The  rows  that  follow  the  first  line  include  the  following  information.  The  first 
column  presents  the  date  and  time  (year/year/month/month/day/day/hour/hour/ 
minute/minute/second/second).  The  next  column  describes  the  meteorological  station, 
and  the  columns  that  follow  provide  the  altitude  (meters  above  ground  level),  wind  speed 
(m/sec),  wind  direction  (in  degrees  from  North),  and  the  standard  deviation  of  the  wind 
direction  (Ce  in  degrees).  Values  of  “-99999”  denote  missing  (or  unmeasured)  data 
points. 


Table  B-1  characterizes  the  input  parameters  used  for  the  Prairie  Grass 
simulations  and  was  derived  from  Reference  B-3.  In  addition,  a  surface  roughness  length 
value  of  0.008  meters  was  specified  for  the  Prairie  Grass  field  trials. 

A  description  of  the  column  headings  and  meanings  of  symbols  appearing  in 
Table  B-1  follows. 

.  Trial  refers  to  the  Prairie  Grass  field  trial  number.  The  cases  marked  by  a 
diamond,  ♦,  in  the  Trial  field  were  eliminated  from  the  comparison  runs 
because  of  missing  data.  Specifications  of  the  reasons  for  the  elimination  of 
individual  cases  are  given  in  Table  B-2. 

.  Year/Month/Day/Time  refers  to  the  date/time  stamp  in  YYYYmmmdd 
hhmm  format. 

.  Experimental  Data  (Raw)  signifies  that  data  was  taken  (or  derived)  from  the 
raw  observations  available  in  the  Prairie  Grass  report. 

.  The  air  temperature,  0o,  is  the  value  at  a  height  of  2  meters  (above  ground 
level)  in  degrees  Celsius  [Ref.  B-1,  Table  8.1,  Volume  II]. 
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Table  B-1.  Table  of  Input  Parameters  Provided  by  LLNL 


Trial 

Year/Month 

Experimental  Data  (Raw) 

Experimental  Data  (Fit) 

Veld 

/Day/Time 

0o 

H0 

U « 

cc 

Ut 

L 

h 

n 

1 956 JUL03 1700 

22.12 

9 

0.19 

-8.6 

871 

2* 

1956JUL03  2100 

23.87 

-0.048 

9 

0.13 

-14 

1042 

3 ♦ 

1956JUL06  0400 

19.83 

3 

73 

-u 

1956JUL06  0700 

17.50 

1 

194 

5 

1956JUL06  2000 

30.17 

0.357 

0 

0.39 

-28 

1100 

0.0042 

6* 

1956JUL06  2300 

30.80 

0.309 

0 

0.44 

-84 

969 

0.0043 

1 

1956JUL10  2000 

30.27 

-0.430 

0.291 

0 

0.31 

-9.8 

2639 

0.0041 

8 

1956JUL10  2300 

31.10 

-0.205 

0.313 

0 

0.31 

-18 

1913 

0.0041 

9 

1956JUL11  1600 

27.39 

-0.331 

0.376 

3 

0.46 

-31 

533 

0.0043 

10 

1956JUL11  1800 

30.45 

-0.382 

0.277 

3 

0.32 

-11 

1064 

0.0041 

11 

1956JUL14  1400 

25.01 

-0.21* 

0 

0.50 

-66 

201 

0.0044 

12 

1956JUL14  1600 

29.67 

-0.38* 

0 

0.52 

-47 

682 

0.0044 

13 

1956JUL23  0200 

20.39 

0.01* 

0.042 

2 

0.09 

3.4 

82 

0.0026 

14 

1956JUL23  0400 

16.25 

0.01* 

0 

0.05 

1.6 

91 

0.0018 

15 

1 956 JUL23 1400 

21.44 

-0.208 

0.290 

0 

0.23 

-7.6 

120 

0.0038 

16 

1 956 JUL23 1600 

25.61 

-0.322 

0.229 

0 

0.24 

-5.2 

1136 

0.0039 

17 

1956JUL24  0200 

27.44 

0.00* 

0.181 

7 

0.21 

48 

141 

0.0037 

18 

1956JUL24  0400 

23.52 

0.109 

1 

0.20 

25 

159 

0.0036 

19 

1 956 JUL25 1700 

28.59 

-0.326 

0.423 

3 

0.39 

-28 

849 

0.0042 

20 

1 956 JUL25 1900 

32.49 

-0.692 

2 

0.60 

-62 

894 

0.0045 

21 

1956JUL26  0400 

28.60 

0.053 

0.342 

10 

0.38 

172 

325* 

0.0042 

22 

1956JUL26  0600 

26.42 

0.092 

0.474 

6 

0.46 

204 

390* 

0.0043 

23 

1956JUL30  0300 

23.40 

0.04* 

0.411 

0 

0.39 

193 

350* 

0.0042 

24 

1956JUL30  0500 

21.89 

0.03* 

0.404 

0 

0.38 

248 

83 

0.0042 

25 

1956AUG01  1900 

23.62 

-0.131 

10 

0.20 

-6.2 

736 

0.0037 

26 

1956AUG02  1800 

29.21 

-0.370 

0.435 

9 

0.43 

-32 

1052 

0.0043 

27 

1956AUG02  2000 

31.43 

-0.445 

0.492 

7 

0.42 

-30 

1396 

0.0043 

28 

1956AUG03  0600 

24.22 

0.151 

0 

0.16 

24 

141 

0.0034 

29* 

1956AUG03  0800 

25.36 

0.23 

36 

122 

0.0038 

30* 

1956AUG03  1900 

33.53 

-0.432 

0 

0.46 

-39 

2055 

0.0043 

31  ♦ 

1956AUG03  2100 

34.25 

-0.280 

2 

0.51 

-67 

1855 

0.0044 

32 

1956AUG07  0200 

22.93 

0.049 

0.067 

1 

0.13 

8.3 

112 

0.0031 

33 

1956AUG07  1900 

28.73 

-0.322 

0.50 

-51 

541 

0.0044 

34 

1956AUG07  2100 

30.19 

-0.450 

2 

0.60 

-76 

1101 

0.0045 

35S* 

22.28 

0.039 

0.244 

2 

0.24 

53 

35 

1956AUG12  0330 

19.73 

0.009 

1 

0.11 

6.8 

64 

0.0029 

36 

1956AUG12  0530 

18.94 

0.028 

0.085 

0 

0.10 

7.8 

120 

0.0028 

37 

1956AUG12  0900 

20.65 

0.324 

3 

0.29 

95 

279 

0.0040 

38 

1956AUG12  1100 

19.82 

0.036 

0.275 

8 

0.28 

99 

345 

0.0040 

39 

1956AUG14  0430 

20.47 

0.043 

0.091 

0 

0.14 

9.8 

167 

0.0032 

40 

1956AUG14  0630 

20.29 

0.034 

0.11 

8.0 

148 

0.0029 

41 

1956AUG14  0900 

20.20 

0.050 

0.253 

0.23 

35 

298 

0.0038 

42 

1956AUG14  1100 

21.49 

0.078 

0.362 

0.37 

120 

252 

0.0042 

43 

1956AUG15  1800 

33.05 

-0.401 

0.366 

10 

0.35 

-16 

1493 

0.0041 

44 

1956AUG15  2000 

35.43 

-0.449 

0.311 

8 

0.40 

-25 

2447 

0.0042 

45 

1956AUG15  2300 

35.04 

-0.104 

0.277 

9 

0.39 

-87 

1990 

0.0042 

46 

1956AUG16  0045 

32.74 

0.064 

0.241 

9 

0.34 

114 

250* 

0.0041 

47 

1956AUG20  1600 

17.73 

-0.21* 

3 

1243 

48S4 

Not  used 

-0.218 

6 

0.21 

-7.8 

48 

1956AUG21  1500 

18.40 

-0.345 

1 

0.51 

-63 

491 

0.0044 

49 

1956AUG21  1700 

23.29 

-0.464 

1 

0.45 

-28 

932 

0.0043 
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Table  B-1.  Table  of  input  parameters  provided  by  LLNL  (cont’d) 


Trial 

Year/Month 

/Day/Time 

Experimental  Data  (Raw) 

Experimental  Data  (Fit) 

Veld 

0o 

H0 

U , 

CC 

Ut 

L 

h 

50 

1956AUG21  2000 

28.64 

-0.542 

1 

0.44 

-26 

1697 

0.0043 

51 

1956AUG21  2130 

29.80 

-0.409 

1 

0.45 

-40 

2332 

0.0043 

52* 

1956AUG24  1717 

24.96 

-0.511 

0 

1154 

53* 

1956AUG25  0200 

17.39 

0.067 

0.051 

0 

0.17 

10 

73 

0.0034 

54 

1956AUG25  0400 

18.68 

0.058 

0.212 

0 

0.24 

40 

285 

0.0038 

55 

1956AUG25  0700 

16.75 

0.077 

0.349 

0 

0.37 

124 

405 

0.0042 

56 

1956AUG25  0900 

15.29 

0.041 

0.199 

0 

0.29 

76 

444 

0.0040 

57 

1956AUG25  2330 

34.11 

-0.082 

0 

0.46 

-194 

1836 

0.0043 

58 

1956AUG26  0130 

23.64 

0.065 

3 

0.11 

6.4 

66 

0.0029 

59 

1956AUG26  0430 

23.60 

0.050 

0.103 

5 

0.14 

11 

248 

0.0032 

60 

1956AUG26  0630 

25.75 

0.068 

0.196 

7 

0.28 

58 

295 

0.0040 

61 

1956AUG27  1700 

31.63 

-0.500 

0.218 

3 

0.51 

-38 

581 

0.0044 

62 

1956AUG27  2000 

30.75 

-0.173 

0.246 

8 

0.34 

-30 

109 

0.0041 

63* 

1956AUG28  0200 

263 

64* 

1956AUG28  0500 

368 

65 

1956AUG30  0130 

24.96 

0.271 

2 

66 

1956AUG30  0330 

20.19 

0.170 

0 

159 

67 

1956AUG30  0630 

20.55 

0.218 

0 

197 

68 

1956AUG30  0830 

20.65 

0.116 

0 

141 

2 

.  The  sensible  heat  flux,  H„,  is  in  cal/cnr-min  and,  in  general,  was  taken  from 
the  Texas  A&M  data  in  Reference  B-1,  Table  9.2,  Volume  I.  The  *  indicates 
that  the  values  were  taken  from  the  University  of  Wisconsin  data  in  Reference 
B-1,  Table  10.1,  Volume  EL 


Friction  velocity,  u*  (m/s),  was  derived  from  ui 
found  in  Table  17.3,  Volume  III  of  Reference  B-2. 


+  V  W 


and  data 


.  The  fractional  cloud  cover,  cc,  are  given  in  tenths  [Ref.  B-1,  Table  3.1, 
Volume  I]. 

.  Experimental  Data  (fit)  signifies  that  data  were  numerically  fitted.  The 
parameter  values  in  these  columns  were  used  in  the  simulations  instead  of  the 
raw  experimental  observations.  This  was  done  in  order  to  resolve  accuracy 
questions  associated  with  the  raw  measured  values.  A  more  reliable  approach 
for  estimating  u *  (friction  velocity  in  m/s)  and  L  (Monin-Obukhov  length 
scale  in  meters)  is  based  on  fits  to  the  temperature  and  wind  speed  profiles 
[Ref.  B-4].  An  almost  complete  set  of  friction  velocity  and  Monin-Obukhov 
lengths  derived  using  this  approach  are  found  in  Reference  B-5.  The 
boundary  layer  height,  h  (meters),  is  determined  from  the  temperature 
soundings.  The  *  in  the  boundary  layer  height  column  signifies  that  the 
standard  default  was  used  [Ref.  B-3]. 

.  Deposition  velocity,  Veld ,  was  determined  according  to  the  following  formula 
[Ref.  B-6]: 


B-4 


where 


Veld 


k  ut./ 

/ log  (zr/z0)+2.6  +  k  ut  rc  -'F 


[exp  (0.598  +  0.39 log  (~z,r  /  L)  - 0.90  [log  (~zr  /  L)f )  L  <  0 

j-5  zr/L  L>  0 

Here  L  is  the  Monin-Obukhov  length  (m),  «*  is  the  friction  velocity  (m/s),  zr 
is  the  reference  height  for  the  deposition  velocity  (m),  andz0  is  the  surface 
roughness  height  (m).  The  canopy  resistance  rc  is  given  by: 
0.0  sec/m  wet  surface 
r  =  \  70.0  sec/m  actively  growing  vegetation 

200.0  sec/m  vegetation  subject  to  small  water  stress 


The  canopy  resistance  was  set  to  200.0  s/m  corresponding  to  the  fairly  arid 
soil  conditions  of  Nebraska  and  the  surface  roughness  height  was  set  to 
0.008m.  The  reference  height  for  deposition  was  set  to  zr=0.5m.  The 
experimental  fit  values  for  friction  velocity,  u *,  were  derived  from  Reference 
B-5  but  were  scaled  by  the  ratio  of  0.40/0.35  to  compensate  for  the  use  of  a 
different  choice  of  the  von  Karman  constant  [Ref.  B-3]. 


Table  B-2  [Ref.  B-3]  describes  the  reasons  for  the  elimination  of  the  trials  that 
had  missing  data.  Five  additional  experiments  were  eliminated  from  the  comparisons,  as 
described  in  Table  B-3  [Ref.  B-3]. 


Table  B-2.  Trials  That  Were  Eliminated  from  the  Comparison  Because  of  Missing  Data 


Trial 

Reason  for  elimination  of  case  from  simulations 

1 

Missing  all  or  part  of  tower  winds  (stations  “SOURCE”  and  “NORTH450”) 

2 

Missing  all  or  part  of  tower  winds  (stations  “SOURCE”  and  “NORTH450”) 

3 

Missing  all  or  part  of  tower  winds  (stations  “SOURCE”  and  “NORTH450”) 

4 

Extremely  diffuse  concentration  measurements 

6 

Missing  all  or  part  of  tower  winds 

29 

Missing  all  or  part  of  tower  winds 

30 

Missing  sampler  concentration  correction  factors 

31 

Missing  sampler  concentration  correction  factors 

35S 

Missing  rawinsonde  data 

48S 

Missing  rawinsonde  data  and  directions  (stations  “SOURCE”  and  “NORTH450”) 

52 

Missing  all  or  part  of  tower  winds 

53 

Missing  all  or  part  of  tower  winds 

63 

Missing  all  or  part  of  tower  winds  and  missing  concentration  data 

64 

Missing  all  or  part  of  tower  winds  and  missing  concentration  data 

Table  B-3.  Additional  Trials  That  Were  Eliminated  from  the  Comparison 


Trial 

Reason  for  elimination  of  case  from  baseline  simulations 

65-68 

Source  height  is  different  for  these  cases  so  no  turbulence  fit  data  was  reported 

47 

Maximum  concentration  <  0.5mg/mA3 
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2.  HPAC  SIMULATION  SETUP 


The  data  discussed  in  section  1  of  this  appendix  were  used  to  produce  two  profiles 
for  the  HPAC  simulations.  HPAC’s  “Upper  Air  Observations”  profile  was  created  from 
the  “Rawinson”  radiosonde  and  “A&M  TWR”  tower  data.  HPAC’s  “Surface 
Observations”  profile  was  derived  from  the  rest  of  the  meteorological  data  shown  in 
Figure  B-l.  Figures  B-2  and  B-3  depict  the  two  types  of  HPAC  meteorological  input 
parameter  files  that  were  created  from  the  information  available  from  the  file  shown  in 
Figure  B-l. 

#  TYPE:  OBSERVATION 

#  TIMEREFERENCE :  UTC 


PROFILE 
7  3 


ID 

YYMMDD 

HOUR  X 

Y 

MOL8 

ZI9 

HOURS  KM 

KM 

M 

M 

Z 

WSPD 

WDIR 

M 

M/S 

DEG 

-9999 . 00 

ID:  000001 

560730 

4 

.83  535.425 

4705 .287 

248 . 00 

83 . 00 

2 . 00 

3.80 

130.00 

290.00 

13.30 

150 . 00 

600.00 

14 .40 

160.00 

930.00 

13 . 70 

190 . 00 

1270 . 00 

14.20 

210 . 00 

ID:  000002 

560730 

5 

.25  534.985 

4705 . 247 

248 . 00 

83 . 00 

0 .25 

3 . 82 

146 . 00 

0 . 50 

4 . 61 

146 . 00 

1 . 00 

5 .21 

146 . 00 

2 . 00 

5 . 86 

146 . 00 

4 . 00 

6 . 62 

146 . 00 

8 . 00 

7 . 64 

146 . 00 

16 . 00 

8 . 65 

146 . 00 

Figure  B-2.  HPAC’s  “Upper  Air  Observations”  Profile  for  Trial  2410 


The  HP  AC-derived  wind  profiles  (described  previously),  the  surface  roughness 
height  (0.008m),  and  the  L  and  h  values  of  Table  B-l  (whose  column  headings  are 
highlighted  in  red)  were  used  to  generate  the  HPAC  predictions.  In  addition,  for  the 


°  The  Monin-Obukhov  Length  (denoted  here  by  MOL)  was  provided  by  LLNL  and  is  shown  in  Table  B- 
4. 

9  The  boundary  layer  mixing  height  (denoted  here  by  ZI)  was  provided  by  LLNL  and  is  shown  in  Table 
B-4 

Station  ID  000001  corresponds  to  “Rawinson”  radiosonde  and  Station  ID  00002  corresponds  to  “A&M 
TWR”.  Note  that  for  “A&M  TWR,”  the  wind  direction  at  all  heights  is  assumed  to  be  identical  to  the 
wind  direction  at  the  height  of  1  meter. 
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HPAC  predictions  that  considered  an  SO2  deposition  mechanism,  the  Veld  values  (whose 


column  heading 

is  highlighted  in  blue)  were  used. 

#  TYPE: 

OBSERVATION 

#  TIMEREFERENCE:  UTC 

SURFACE 

10 

ID  YYMMDD 

HOUR  X 

Y 

Z 

WSPD 

WDIR 

MOL  ZI 

HOURS  KM 

KM 

M 

M/S 

DEG 

M 

M 

-9999 . 00 

KSRC  560730 

5.17  535.200 

4704.427 

2 . 00 

6.22 

141 . 00 

248 . 00 

83 . 00 

KNRT  560730 

5.17  535.170 

4704 . 877 

2 . 00 

5 . 76 

142 . 00 

248 . 00 

83 . 00 

KMTT  560730  5.25  535.109  4705.251  2.00  -9999.00  135.40  248.00  83.00 

Figure  B-3.  HPAC’s  “Surface  Observations”  Profile  for  Trial  2411 


1 1  Station  IDs  “KSRC”,  “KNRT”  and  “KMTT”  denote  the  stations  “SOURCE”,  “NORTH450”  and  “MIT 
3L”  from  Figure  B-l,  respectively. 
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APPENDIX  C 

HPAC  PREDICTIONS  COMPARED  TO  PRAIRIE  GRASS  TRIALS 


This  appendix  presents  graphical  comparison  of  HPAC  predictions  with  no  SO2 
surface  deposition  to  Prairie  Grass  field  trials  [Ref.  C-l].  Vertical  plot  units  are  dosage 
units  of  mg-sec/m  .  Horizontal  plot  units  are  sampler  numbers  as  presented  in  the  Prairie 
Grass  field  trials  with  sampler  number  1  oriented  to  the  west,  the  middle  sampler  (45  or 
90)  oriented  to  the  north,  and  the  last  sampler  (91  or  181)  oriented  to  the  east  of  the  SO2 
gas  release  source.  Only  data  values  greater  than  the  cutoff  threshold  of  3  mg-sec/m 
(0.005  mg/m  )  are  presented  for  both  field  trial  data  and  HPAC  predictions.  This  cutoff 
threshold  value  corresponds  to  a  minimum  value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  HPAC  predictions  and  Prairie  Grass  field  trial  observations  are 
presented  on  both  linear  and  logarithmic  dosage  scales.  Each  graphical  comparison 
consists  of  five  plots  (one  for  each  arc)  with  the  top  plot  depicting  the  50-meter  arc,  the 
second  plot  depicting  the  100-meter  arc,  the  third  plot  depicting  the  200-meter  arc,  and  so 
on.  The  last  panel  (just  above  the  figure  caption)  contains  information  about  the  data 
files  used  to  produce  these  plots.  The  Prairie  Grass  field  trial  file  name  contains  a 
two-digit  number  corresponding  to  the  trial  number.  The  HPAC  prediction  file  name 
contains  the  moniker  “nodeposition,”  denoting  that  SO2  surface  deposition  was  not 
considered  in  these  predictions,  and  a  one-  or  two-digit  number  reflecting  the  Prairie 
Grass  field  trial  being  predicted.  Odd-numbered  pages  contain  figures  on  the  linear 
dosage  scale  while  even-numbered  pages  contain  figures  on  the  logarithmic  dosage  scale. 

The  meanings  of  the  colors  used  in  the  plots  are  described  below: 


.  Been 

.  Yellow  —  prediction  dosages  that  are  higher  than  the  trial 

.  White  —  trial  sampler  observation  is  missing. 

These  shadings  correspond  to  AE2  as  described  in  Chapter  2. 

Samplers  with  missing  values,  or  spurious  maximum  that  were  fixed  by  moving 
the  decimal  point,  are  noted  in  the  figure  captions  [Ref.  C-2]  (see  also  Appendix  K). 
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Irwin  stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure 
captions.  These  stability  category  assignments  are  based  on  Reference  C-3. 
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Figure  C-1a.  HPAC  Predictions  to  Trial  5  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-lb.  HPAC  Prediction  to  Trial  5  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  C-2a.  HPAC  Prediction  to  Trial  7  on  Linear  Scale:  Stability  Category  is  1 
(Value  for  Sampler  45  of  100-Meter  Arc  is  Missing) 
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Figure  C-2b.  HPAC  Prediction  to  Trial  7  on  Logarithmic  Scale:  Stability  Category  is  1 
(Value  for  Sampler  45  of  100-Meter  Arc  is  Missing) 
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Figure  C-3a.  HPAC  Prediction  to  Trial  8  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-3b.  HPAC  Prediction  to  Trial  8  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  C-4a.  HPAC  Prediction  to  Trial  9  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-4b.  HPAC  Prediction  to  Trial  9  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  C-5a.  HPAC  Prediction  to  Trial  10  on  Linear  Scale:  Stability  Category  is  1 
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Figure  C-5b.  HPAC  Prediction  to  Trial  10  on  Logarithmic  Scale: 
Stability  Category  is  1 
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Figure  C-6a.  HPAC  Prediction  to  Trial  11  on  Linear  Scale: 

Stability  Category  is  3  (Value  for  Sampler  90  of  800-Meter  Arc  is  Considered  “Spurious” 

and  is  Fixed  by  Moving  Decimal  Point) 
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PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_<grass_tracer_£xperiment_1 1  .txt 

Prediction  File:  HPAC\nodeposition\pg_1 1_novd.out 


Figure  C-6b.  HPAC  Prediction  to  Trial  11  on  Logarithmic  Scale: 

Stability  Category  is  3  (Value  for  Sampler  90  of  800-Meter  Arc  is  Considered  “Spurious” 

and  is  Fixed  by  Moving  Decimal  Point) 
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Figure  C-7a.  HPAC  Prediction  to  Trial  12  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-7b.  HPAC  Prediction  to  Trial  12  on  Logarithmic  Scale: 
Stability  Category  is  3 
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Figure  C-8a.  HPAC  Prediction  to  Trial  13  on  Linear  Scale:  Stability  Category  is  7 
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Figure  C-8b.  HPAC  Prediction  to  Trial  13  on  Logarithmic  Scale: 
Stability  Category  is  7 
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Figure  C-9a.  HPAC  Prediction  to  Trial  14  on  Linear  Scale:  Stability  Category  is  7 
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Figure  C-9b.  HPAC  Prediction  to  Trial  14  on  Logarithmic  Scale: 
Stability  Category  is  7 
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Figure  C-10a.  HPAC  Prediction  to  Trial  15  on  Linear  Scale:  Stability  Category  is  1 
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Figure  C-IOb.  HPAC  Prediction  to  Trial  15  on  Logarithmic  Scale: 
Stability  Category  is  1 
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Figure  C-11a.  HPAC  Prediction  to  Trial  16  on  Linear  Scale:  Stability  Category  is  1  (Value 
for  Sampler  63  of  200-Meter  Arc  is  Considered  “Spurious”  and  is  Fixed  by  Moving  Decimal 

Point) 
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Figure  C-llb.  HPAC  Prediction  to  Trial  16  on  Logarithmic  Scale: 

Stability  Category  is  1  (Value  for  Sampler  63  of  200-Meter  Arc  is  Considered  “Spurious” 

and  is  Fixed  by  Moving  Decimal  Point) 
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Figure  C-12a.  HPAC  Prediction  to  Trial  17  on  Linear  Scale:  Stability  Category  is  5 
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Figure  C-12b.  HPAC  Prediction  to  Trial  17  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-13a.  HPAC  Prediction  to  Trial  18  on  Linear  Scale:  Stability  Category  is  5 
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Figure  C-13b.  HPAC  Prediction  to  Trial  18  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-14a.  HPAC  Prediction  to  Trial  19  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-14b.  HPAC  Prediction  to  Trial  19  on  Logarithmic  Scale: 
Stability  Category  is  2 
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Figure  C-15a.  HPAC  Prediction  to  Trial  20  on  Linear  Scale:  Stability  Category  is  3 


C-31 


DOSAGE 


ARC50 


LlJ 


a 


o 

o 


% 


o 

o 


o 

D 


20  40  60  80 


Sompler  number 

ARC 4 00 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_g rass_tracer_Experiment_20.txt 

Prediction  File:  HPAC\nodeposition\pg_20_novd.out 


Figure  C-15b.  HPAC  Prediction  to  Trial  20  on  Logarithmic  Scale: 
Stability  Category  is  3 
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PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_<grass_tracer_Experiment_2l  .txt 
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Figure  C-16a.  HPAC  Prediction  to  Trial  21  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-16b.  HPAC  Prediction  to  Trial  21  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-17a.  HPAC  Prediction  to  Trial  22  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-17b.  HPAC  Prediction  to  Trial  22  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-18a.  HPAC  Prediction  to  Trial  23  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-18b.  HPAC  Prediction  to  Trial  23  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-19a.  HPAC  Prediction  to  Trial  24  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-19b.  HPAC  Prediction  to  Trial  24  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-20a.  HPAC  Prediction  to  Trial  25  on  Linear  Scale:  Stability  Category  is  1  (Values 
for  Samplers  47  55  of  400-Meter  Arc  and  64,  75,  77,  91  of  800-Meter  Arc  are  Missing) 
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Figure  C-20b.  HPAC  Prediction  to  Trial  25  on  Logarithmic  Scale: 

Stability  Category  is  1  (Values  for  Samplers  47  55  of  400-Meter  Arc  and  64,  75,  77,  91  of 

800-Meter  Arc  are  Missing) 
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Figure  C-21a.  HPAC  Prediction  to  Trial  26  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-21b.  HPAC  Prediction  to  Trial  26  on  Logarithmic  Scale: 
Stability  Category  is  2 
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PG  Trial  File:  pr_grass_tracer_Experiment_27.txt 
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Figure  C-22a.  HPAC  Prediction  to  Trial  27  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-22b.  HPAC  Prediction  to  Trial  27  on  Logarithmic  Scale: 
Stability  Category  is  2 
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PG  Trial  File:  pr_grass_tracer_Experiment_28.txt 

Prediction  File:  HPAC\nodep09ition\pg_28_novd.out 


Figure  C-23a.  HPAC  Prediction  to  Trial  28  on  Linear  Scale:  Stability  Category  is  5 


C-47 


DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


o 

□ 


50  100  150 

Sampler  number 


20 


10000F 


1000  i- 


40  60 

Sampler  number 

ARC 800 


80 


PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_grass_tracer_Experiment_28.txt 

Prediction  File:  MPAC\nodeposition\pg_28_novd.out 


Figure  C-23b.  HPAC  Prediction  to  Trial  28  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-24a.  HPAC  Prediction  to  Trial  32  on  Linear  Scale:  Stability  Category  is  6 
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Figure  C-24b.  HPAC  Prediction  to  Trial  32  on  Logarithmic  Scale: 
Stability  Category  is  6 
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Figure  C-25a.  HPAC  Prediction  to  Trial  33  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-25b.  HPAC  Prediction  to  Trial  33  on  Logarithmic  Scale: 
Stability  Category  is  3 
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Figure  C-26a.  HPAC  Prediction  to  Trial  34  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-26b.  HPAC  Prediction  to  Trial  34  on  Logarithmic  Scale: 
Stability  Category  is  3 
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Figure  C-27a.  HPAC  Prediction  to  Trial  35  on  Linear  Scale:  Stability  Category  is  6 
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Figure  C-27b.  HPAC  Prediction  to  Trial  35  on  Logarithmic  Scale: 
Stability  Category  is  6 
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Figure  C-28a.  HPAC  Prediction  to  Trial  36  on  Linear  Scale:  Stability  Category  is  6 
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Figure  C-28b.  HPAC  Prediction  to  Trial  36  on  Logarithmic  Scale: 
Stability  Category  is  6 
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Figure  C-29a.  HPAC  Prediction  to  Trial  37  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-29b.  HPAC  Prediction  to  Trial  37  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-30a.  HPAC  Prediction  to  Trial  38  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-30b.  HPAC  Prediction  to  Trial  38  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-31a.  HPAC  Prediction  to  Trial  39  on  Linear  Scale:  Stability  Category  is  6  (Values 
for  Samplers  40-49,  56-58  of  800-Meter  Arc  are  Missing) 
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Figure  C-31b.  HPAC  Prediction  to  Trial  39  on  Logarithmic  Scale:  Stability  Category  is  6 
(Values  for  Samplers  40-49,  56-58  of  800-Meter  Arc  are  Missing) 
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Figure  C-32a.  HPAC  Prediction  to  Trial  40  on  Linear  Scale:  Stability  Category  is  6  (Values 
for  Samplers  39-45  of  400-Meter  Arc  are  Missing) 
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Figure  C-32b.  HPAC  Prediction  to  Trial  40  on  Logarithmic  Scale:  Stability  Category  is  6 
(Values  for  Samplers  39-45  of  400-Meter  Arc  are  Missing) 
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Figure  C-33a.  HPAC  Prediction  to  Trial  41  on  Linear  Scale:  Stability  Category  is  5 
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Figure  C-33b.  HPAC  Prediction  to  Trial  41  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-34a.  HPAC  Prediction  to  Trial  42  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-34b.  HPAC  Prediction  to  Trial  42  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-35a.  HPAC  Prediction  to  Trial  43  on  Linear  Scale:  Stability  Category  is  1 
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Figure  C-35b.  HPAC  Prediction  to  Trial  43  on  Logarithmic  Scale: 
Stability  Category  is  1 
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Figure  C-36a.  HPAC  Prediction  to  Trial  44  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-36b.  HPAC  Prediction  to  Trial  44  on  Logarithmic  Scale: 
Stability  Category  is  2 
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Figure  C-37a.  HPAC  Prediction  to  Trial  45  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-37b.  HPAC  Prediction  to  Trial  45  on  Logarithmic  Scale: 
Stability  Category  is  3 
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Figure  C-38a.  HPAC  Prediction  to  Trial  46  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-38b.  HPAC  Prediction  to  Trial  46  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-39a.  HPAC  Prediction  to  Trial  48  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-39b.  HPAC  Prediction  to  Trial  48  on  Logarithmic  Scale: 
Stability  Category  is  3 
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Figure  C-40a.  HPAC  Prediction  to  Trial  49  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-40b.  HPAC  Prediction  to  Trial  49  on  Logarithmic  Scale: 
Stability  Category  is  2 
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Figure  C-41a.  HPAC  Prediction  to  Trial  50  on  Linear  Scale:  Stability  Category  is  2  (Values 
for  Samplers  of  200-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 
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Figure  C-41b.  HPAC  Prediction  to  Trial  50  on  Logarithmic  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  200-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 


C-84 


ARC50 


i.5v  ur 


g  i.ov io' 
o 

°  5.0*1 04 


4.0*10  b 


2000 


40 

Sampler  number 

ARC 100 


40  60 

Sampler  number 

ARC  2  00 


40 

Sampler  number 
ARC 4 00 


40  60 

Sampler  number 

ARC 800 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_grass_tracer_£xperiment_5l  .txt 

Prediction  File:  HPAC\nodeposition\pg_51_novd.out 


Figure  C-42a.  HPAC  Prediction  to  Trial  51  on  Linear  Scale:  Stability  Category  is  2 
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Figure  C-42b.  HPAC  Prediction  to  Trial  51  on  Logarithmic  Scale: 
Stability  Category  is  2 
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Figure  C-43a.  HPAC  Prediction  to  Trial  54  on  Linear  Scale:  Stability  Category  is  5 
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Figure  C-43b.  HPAC  Prediction  to  Trial  54  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-44a.  HPAC  Prediction  to  Trial  55  on  Linear  Scale:  Stability  Category  is  4 


C-89 


30VS0Q  30VS0Q  30VS0Q 


ARC50 


10000 s 


40 

Sampler  number 

ARC 100 


40  60 

Sompler  number 
ARC 4 00 


40  60 

Sompler  number 

ARC 800 


100 

Sompler  number 


PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_<g rass_tracer_Experiment_55.txt 

Prediction  File:  HPAC\nodep09ition\pg_55_novd.out 


Figure  C-44b.  HPAC  Prediction  to  Trial  55  on  Logarithmic  Scale: 
Stability  Category  is  4 
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Figure  C-45a.  HPAC  Prediction  to  Trial  56  on  Linear  Scale:  Stability  Category  is  4 
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Figure  C-45b.  HPAC  Prediction  to  Trial  56  on  Logarithmic  Scale: 
Stability  Category  is  4 


C-92 


ARC50 


2,0*1 0  - 

o 

»  1.0*1  o5 - 

o 

D 

5.0*10’ 


1.5*10 


g  1.0*10’  F- 
o 

D  5.0*1 0“'  F- 


5000  | 
4000  I- 
9  3000  i- 

s 

g  2000  i- 

1000  I- 


1200  E- 
1000  E- 
5  soo  =- 


s 

o 

o 


600  E- 
400  E- 
200  E- 


50 


40 

Sampler  number 

ARC 100 


ARC  2  00 


Sompler  number 
ARC 4 00 


Sompler  number 

ARC 800 


100 

Sompler  number 


150 


PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_g rass_tracer_Experiment_57.txt 

Prediction  File:  MPAC\nodeposition\pgJ57_navd.out 


Figure  C-46a.  HPAC  Prediction  to  Trial  57  on  Linear  Scale:  Stability  Category  is  3  (Values 
for  Sampler  of  100-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 
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Figure  C-46b.  HPAC  Prediction  to  Trial  57  on  Logarithmic  Scale:  Stability  Category  is  3 
(Values  for  Sampler  of  100-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 
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Figure  C-47a.  HPAC  Prediction  to  Trial  58  on  Linear  Scale:  Stability  Category  is  6 
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Figure  C-47b.  HPAC  Prediction  to  Trial  58  on  Logarithmic  Scale: 
Stability  Category  is  6 
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Figure  C-48a.  HPAC  Prediction  to  Trial  59  on  Linear  Scale:  Stability  Category  is  5 


C-97 


DOSAGE  DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_g rass_tracer_Experiment_59.txt 

Prediction  File:  HPAC\nodeposition\pg_59_novd.out 


Figure  C-48b.  HPAC  Prediction  to  Trial  59  on  Logarithmic  Scale: 
Stability  Category  is  5 
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Figure  C-49a.  HPAC  Prediction  to  Trial  60  on  Linear  Scale:  Stability  Category  is  5 
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Figure  C-49b.  HPAC  Prediction  to  Trial  60  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  C-50a.  HPAC  Prediction  to  Trial  61  on  Linear  Scale:  Stability  Category  is  3 
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Figure  C-50b.  HPAC  Prediction  to  Trial  61  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  C-51a.  HPAC  Prediction  to  Trial  62  on  Linear  Scale:  Stability  Category  is  2  (Values 
for  Samplers  of  50-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 
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Figure  C-51b.  HPAC  Prediction  to  Trial  62  on  Logarithmic  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  50-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  C-2]) 
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APPENDIX  D 

NARAC  PREDICTIONS  COMPARED  TO  PRAIRIE  GRASS  TRIALS 


This  appendix  presents  graphical  comparison  of  NARAC  predictions  with  no  SO2 
surface  deposition  to  Prairie  Grass  field  trials  [Ref.  D-l].  Vertical  plot  units  are  dosage 
units  of  mg-sec/m  .  Horizontal  plot  units  are  sampler  numbers  as  presented  in  the  Prairie 
Grass  field  trials  with  sampler  number  1  oriented  to  the  west,  the  middle  sampler  (45  or 
90)  oriented  to  the  north,  and  the  last  sampler  (91  or  181)  oriented  to  the  east  of  the  SO2 
gas  release  source.  Only  data  values  greater  than  the  cutoff  threshold  of  3  mg-sec/m 

a 

(0.005  mg/m  )  are  presented  for  both  field  trial  data  and  NARAC  predictions.  This  cutoff 
threshold  value  corresponds  to  a  minimum  value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  NARAC  predictions  and  Prairie  Grass  field  trial  observations  are 
presented  on  both  linear  and  logarithmic  dosage  scales.  Each  graphical  comparison 
consists  of  five  plots  (one  for  each  arc)  with  the  top  plot  depicting  the  50-meter  arc,  the 
second  plot  depicting  the  100-meter  arc,  the  third  plot  depicting  the  200-meter  arc,  and  so 
on.  The  last  panel  (just  above  the  figure  caption)  contains  information  about  the  data 
files  used  to  produce  these  plots.  The  Prairie  Grass  field  trial  file  name  contains  a 
two-digit  number  corresponding  to  the  trial  number.  The  NARAC  prediction  file  name 
contains  the  moniker  “nodeposition,”  denoting  that  SO2  surface  deposition  was  not 
considered  in  these  predictions,  and  a  one-  or  two-digit  number  reflecting  the  Prairie 
Grass  field  trial  being  predicted.  Odd-numbered  pages  contain  figures  on  the  linear 
dosage  scale  while  even-numbered  pages  contain  figures  on  the  logarithmic  dosage  scale. 

The  meanings  of  the  colors  used  in  the  plots  are  described  below: 


Green  — 

trial  dosages  that  overlap  with  the  prediction 

Yellow  — 

prediction  dosages  that  are  higher  than  the  trial 

.  White  —  trial  sampler  observation  is  missing. 

These  shadings  correspond  to  AE2  as  described  in  Chapter  2. 

Samplers  with  missing  values,  or  spurious  maximum  that  were  fixed  by  moving 
the  decimal  point,  are  noted  in  the  figure  captions  [Ref.  D-2]  (see  also  Appendix  K). 
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Irwin  stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure 
captions.  These  stability  category  assignments  are  based  on  Reference  D-3. 
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Figure  D-1a.  NARAC  Predictions  to  Trial  5  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-lb.  NARAC  Prediction  to  Trial  5  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-2a.  NARAC  Prediction  to  Trial  7  on  Linear  Scale:  Stability  Category  is  1 
(Value  for  Sampler  45  of  100-Meter  Arc  is  Missing) 
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Figure  D-2b.  NARAC  Prediction  to  Trial  7  on  Logarithmic  Scale:  Stability  Category  is  1 
(Value  for  Sampler  45  of  100-Meter  Arc  is  Missing) 
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Figure  D-3a.  NARAC  Prediction  to  Trial  8  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-3b.  NARAC  Prediction  to  Trial  8  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-4a.  NARAC  Prediction  to  Trial  9  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-4b.  NARAC  Prediction  to  Trial  9  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-5a.  NARAC  Prediction  to  Trial  10  on  Linear  Scale:  Stability  Category  is  1 
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Figure  D-5b.  NARAC  Prediction  to  Trial  10  on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  D-6a.  NARAC  Prediction  to  Trial  11  on  Linear  Scale:  Stability  Category  is  3 
(Value  for  Sampler  90  of  800-Meter  Arc  is  Considered  “Spurious”  and  is  Fixed  by  Moving 

Decimal  Point) 
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Figure  D-6b.  NARAC  Prediction  to  Trial  11  on  Logarithmic  Scale:  Stability  Category  is  3 
(Value  for  Sampler  90  of  800-Meter  Arc  is  Considered  “Spurious”  and  is  Fixed  by  Moving 

Decimal  Point) 
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Figure  D-7a.  NARAC  Prediction  to  Trial  12  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-7b.  NARAC  Prediction  to  Trial  12  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-8a.  NARAC  Prediction  to  Trial  13  on  Linear  Scale:  Stability  Category  is  7 
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Figure  D-8b.  NARAC  Prediction  to  Trial  13  on  Logarithmic  Scale:  Stability  Category  is  7 
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Figure  D-9a.  NARAC  Prediction  to  Trial  14  on  Linear  Scale:  Stability  Category  is  7 
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Figure  D-9b.  NARAC  Prediction  to  Trial  14  on  Logarithmic  Scale:  Stability  Category  is  7 
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Figure  D-10a.  NARAC  Prediction  to  Trial  15  on  Linear  Scale:  Stability  Category  is  1 
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Figure  D-IOb.  NARAC  Prediction  to  Trial  15  on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  D-11a.  NARAC  Prediction  to  Trial  16  on  Linear  Scale:  Stability  Category  is  1  (Value 
for  Sampler  63  of  200-Meter  Arc  is  Considered  “Spurious”  and  is  Fixed  by  Moving  Decimal 

Point) 
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Figure  D-llb.  NARAC  Prediction  to  Trial  16  on  Logarithmic  Scale:  Stability  Category  is  1 
(Value  for  Sampler  63  of  200-Meter  Arc  is  Considered  “Spurious”  and  is  Fixed  by  Moving 

Decimal  Point) 
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Figure  D-12a.  NARAC  Prediction  to  Trial  17  on  Linear  Scale:  Stability  Category  is  5 
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Figure  D-12b.  NARAC  Prediction  to  Trial  17  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-13a.  NARAC  Prediction  to  Trial  18  on  Linear  Scale:  Stability  Category  is  5 
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Figure  D-13b.  NARAC  Prediction  to  Trial  18  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-14a.  NARAC  Prediction  to  Trial  19  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-14b.  NARAC  Prediction  to  Trial  19  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-15a.  NARAC  Prediction  to  Trial  20  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-15b.  NARAC  Prediction  to  Trial  20  on  Logarithmic  Scale:  Stability  Category  is  3 


D-32 


DOSAGE  DOSAGE 


ARC50 


100 

Sampler  number 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_<grass_tracer_Experiment_2l  .txt 

Prediction  File:  ARAC\nodeposition\pg_21_novd.orac 


Figure  D-16a.  NARAC  Prediction  to  Trial  21  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-16b.  NARAC  Prediction  to  Trial  21  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-17a.  NARAC  Prediction  to  Trial  22  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-17b.  NARAC  Prediction  to  Trial  22  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-18a.  NARAC  Prediction  to  Trial  23  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-18b.  NARAC  Prediction  to  Trial  23  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-19a.  NARAC  Prediction  to  Trial  24  on  Linear  Scale:  Stability  Category  is  4 


D-39 


ARC50 


20  40  60  80 


Sampler  number 


50  100  150 

Sampler  number 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_rgrass_tracer_Experiment_24-.txt 

Prediction  File:  ARAC\nodeposition\pg_24_novd.orac 


Figure  D-19b.  NARAC  Prediction  to  Trial  24  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-20a.  NARAC  Prediction  to  Trial  25  on  Linear  Scale:  Stability  Category  is  1 
(Values  for  Samplers  47  55  of  400-Meter  Arc  and  64,  75,  77,  91  of  800-Meter  Arc  are 

Missing) 
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Figure  D-20b.  NARAC  Prediction  to  Trial  25  on  Logarithmic  Scale:  Stability  Category  is  1 
(Values  for  Samplers  47  55  of  400-Meter  Arc  and  64,  75,  77,  91  of  800-Meter  Arc  are 

Missing) 
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Figure  D-21a.  NARAC  Prediction  to  Trial  26  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-21b.  NARAC  Prediction  to  Trial  26  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-22a.  NARAC  Prediction  to  Trial  27  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-22b.  NARAC  Prediction  to  Trial  27  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-23a.  NARAC  Prediction  to  Trial  28  on  Linear  Scale:  Stability  Category  is  5 
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Figure  D-23b.  NARAC  Prediction  to  Trial  28  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-24a.  NARAC  Prediction  to  Trial  32  on  Linear  Scale:  Stability  Category  is  6 
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Figure  D-24b.  NARAC  Prediction  to  Trial  32  on  Logarithmic  Scale:  Stability  Category  is  6 
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Figure  D-25a.  NARAC  Prediction  to  Trial  33  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-25b.  NARAC  Prediction  to  Trial  33  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-26a.  NARAC  Prediction  to  Trial  34  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-26b.  NARAC  Prediction  to  Trial  34  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-27a.  NARAC  Prediction  to  Trial  35  on  Linear  Scale:  Stability  Category  is  6 
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Figure  D-27b.  NARAC  Prediction  to  Trial  35  on  Logarithmic  Scale:  Stability  Category  is  6 


D-56 


DOSAGE  DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


20 


40 

Sampler  number 

ARC 100 


60 


80 


40 

Sampler  number 

ARC  2  00 


40 

Sampler  number 

ARC 800 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_rgrass_tracer_Experiment_36.txt 

Prediction  File:  ARAC\nodeposition\pg_J6_novd.orac 


Figure  D-28a.  NARAC  Prediction  to  Trial  36  on  Linear  Scale:  Stability  Category  is  6 
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Figure  D-28b.  NARAC  Prediction  to  Trial  36  on  Logarithmic  Scale:  Stability  Category  is  6 
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Figure  D-29a.  NARAC  Prediction  to  Trial  37  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-29b.  NARAC  Prediction  to  Trial  37  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-30a.  NARAC  Prediction  to  Trial  38  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-30b.  NARAC  Prediction  to  Trial  38  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-31a.  NARAC  Prediction  to  Trial  39  on  Linear  Scale:  Stability  Category  is  6 
(Values  for  Samplers  40-49,  56-58  of  800-Meter  Arc  are  Missing) 
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Figure  D-31b.  NARAC  Prediction  to  Trial  39  on  Logarithmic  Scale:  Stability  Category  is  6 
(Values  for  Samplers  40-49,  56-58  of  800-Meter  Arc  are  Missing) 
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Figure  D-32a.  NARAC  Prediction  to  Trial  40  on  Linear  Scale:  Stability  Category  is  6 
(Values  for  Samplers  39-45  of  400-Meter  Arc  are  Missing) 
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Figure  D-32b.  NARAC  Prediction  to  Trial  40  on  Logarithmic  Scale:  Stability  Category  is  6 
(Values  for  Samplers  39-45  of  400-Meter  Arc  are  Missing) 
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Figure  D-33a.  NARAC  Prediction  to  Trial  41  on  Linear  Scale:  Stability  Category  is  5 
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Figure  D-33b.  NARAC  Prediction  to  Trial  41  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-34a.  NARAC  Prediction  to  Trial  42  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-34b.  NARAC  Prediction  to  Trial  42  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-35a.  NARAC  Prediction  to  Trial  43  on  Linear  Scale:  Stability  Category  is  1 
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Figure  D-35b.  NARAC  Prediction  to  Trial  43  on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  D-36a.  NARAC  Prediction  to  Trial  44  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-36b.  NARAC  Prediction  to  Trial  44  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-37a.  NARAC  Prediction  to  Trial  45  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-37b.  NARAC  Prediction  to  Trial  45  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-38a.  NARAC  Prediction  to  Trial  46  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-38b.  NARAC  Prediction  to  Trial  46  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-39a.  NARAC  Prediction  to  Trial  48  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-39b.  NARAC  Prediction  to  Trial  48  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-40a.  NARAC  Prediction  to  Trial  49  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-40b.  NARAC  Prediction  to  Trial  49  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-41a.  NARAC  Prediction  to  Trial  50  on  Linear  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  200-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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Figure  D-41b.  NARAC  Prediction  to  Trial  50  on  Logarithmic  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  200-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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Figure  D-42a.  NARAC  Prediction  to  Trial  51  on  Linear  Scale:  Stability  Category  is  2 
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Figure  D-42b.  NARAC  Prediction  to  Trial  51  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  D-43a.  NARAC  Prediction  to  Trial  54  on  Linear  Scale:  Stability  Category  is  5 


D-87 


DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


50  100  150 

Sampler  number 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_rgrass_tracer_Experiment_54-.txt 

Prediction  File:  ARAC\nodeposition\pgJ54_novd.arac 


Figure  D-43b.  NARAC  Prediction  to  Trial  54  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-44a.  NARAC  Prediction  to  Trial  55  on  Linear  Scale:  Stability  Category  is  4 


D-89 


ARC50 


20  40  60  80 


Sampler  number 


50  100  150 

Sampler  number 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_rgrass_tracer_Experiment_55.txt 

Prediction  File:  ARAC\nodeposition\pg_55_novd.orac 


Figure  D-44b.  NARAC  Prediction  to  Trial  55  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-45a.  NARAC  Prediction  to  Trial  56  on  Linear  Scale:  Stability  Category  is  4 
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Figure  D-45b.  NARAC  Prediction  to  Trial  56  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  D-46a.  NARAC  Prediction  to  Trial  57  on  Linear  Scale:  Stability  Category  is  3 
(Values  for  Sampler  of  100-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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Figure  D-46b.  NARAC  Prediction  to  Trial  57  on  Logarithmic  Scale:  Stability  Category  is  3 
(Values  for  Sampler  of  100-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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Figure  D-47a.  NARAC  Prediction  to  Trial  58  on  Linear  Scale:  Stability  Category  is  6 
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Figure  D-47b.  NARAC  Prediction  to  Trial  58  on  Logarithmic  Scale:  Stability  Category  is  6 
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Figure  D-48a.  NARAC  Prediction  to  Trial  59  on  Linear  Scale:  Stability  Category  is  5 


D-97 


DOSAGE  DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


PG  Observation  to  Prediction  Comparison 

PG  Trial  File:  pr_<g rass_tracer_Experiment_59.txt 

Prediction  File:  ARAC\nodeposition\pg_59_novd.orac 


Figure  D-48b.  NARAC  Prediction  to  Trial  59  on  Logarithmic  Scale:  Stability  Category  is  5 


D-98 


DOSAGE 


ARC50 


20  40  60  80 

Sampler  number 

ARC 100 


ARC  2  00 


ARC 800 


50  100  150 

Sampler  number 


PG  Observation  ta  Prediction  Comparison 

PG  Trial  File:  pr_rgrass_tracer_Experimentj60.txt 

Prediction  File:  ARAC\nodeposition\pg_60_novd.crac 


Figure  D-49a.  NARAC  Prediction  to  Trial  60  on  Linear  Scale:  Stability  Category  is  5 
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Figure  D-49b.  NARAC  Prediction  to  Trial  60  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  D-50a.  NARAC  Prediction  to  Trial  61  on  Linear  Scale:  Stability  Category  is  3 
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Figure  D-50b.  NARAC  Prediction  to  Trial  61  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  D-51a.  NARAC  Prediction  to  Trial  62  on  Linear  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  50-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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Figure  D-51b.  NARAC  Prediction  to  Trial  62  on  Logarithmic  Scale:  Stability  Category  is  2 
(Values  for  Samplers  of  50-Meter  Arc  are  Replaced  by  Missing  Values  [Ref.  D-2]) 
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APPENDIX  E 

COMPARISONS  OF  HPAC  AND  NARAC  PREDICTIONS 


This  appendix  presents  graphical  comparison  of  HPAC  and  NARAC  predictions 

a 

with  no  SOo  surface  deposition.  Vertical  plot  units  are  dosage  units  of  mg-sec/m  .  The 
horizontal  plot  units  are  sampler  numbers  as  presented  in  the  Prairie  Grass  field  trials 
[Ref.  E-l].  Sampler  number  1  is  oriented  to  the  west,  the  middle  sampler  (45  or  90)  is 
oriented  to  the  north,  and  the  last  sampler  (91  or  181)  is  oriented  to  the  east  of  the  SO2 
gas  release  source.  Only  data  values  greater  than  the  cutoff  threshold  of  3  mg-sec/m 
(0.005  mg/m  )  are  presented.  This  cutoff  threshold  value  corresponds  to  a  minimum 
value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  HPAC  and  NARAC  predictions  are  presented  on  both  linear  and 
logarithmic  dosage  scales.  Each  graphical  comparison  consists  of  five  plots  (one  for  each 
arc)  with  the  top  plot  depicting  the  50-meter  arc,  the  second  plot  depicting  the  100-meter 
arc,  the  third  plot  depicting  the  200-meter  arc,  and  so  on.  The  last  panel  (just  above  the 
figure  caption)  contains  information  about  the  data  files  used  to  produce  these  plots.  The 
Prairie  Grass  field  trial  file  name  contains  a  two-digit  number  corresponding  to  the  trial 
number.  Both  HPAC  and  NARAC  prediction  file  names  contain  the  moniker 
“nodeposition,”  denoting  that  SO2  surface  deposition  was  not  considered  in  these 
predictions,  and  a  one-  or  two-digit  number  reflecting  the  Prairie  Grass  field  trial  being 
predicted.  Even-numbered  pages  contain  figures  on  the  linear  dosage  scale  while  odd- 
numbered  pages  contain  figures  on  the  logarithmic  dosage  scale. 

The  meanings  of  the  colors  used  in  the  plots  are  described  below: 


HPAC  dosages  that  are  higher  than  NARAC  dosages 


.  Turquoise  —  HPAC  dosages  that  overlap  with  NARAC  dosages 

.  Brown  —  NARAC  dosages  that  are  higher  than  HPAC  dosages. 

The  dashed  lines  depict  dosage  values  actually  obtained  in  the  field  trial.  Irwin 
stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure  captions. 
These  stability  category  assignments  are  based  on  Reference  E-2. 
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Figure  E-1a.  HPAC  and  NARAC  Predictions  to  Trial  5  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-lb.  HPAC  and  NARAC  Predictions  to  Trial  5  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-2a.  HPAC  and  NARAC  Predictions  to  Trial  7  on  Linear  Scale: 

Stability  Category  is  1 
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Figure  E-2b.  HPAC  and  NARAC  Predictions  to  Trial  7  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-3a.  HPAC  and  NARAC  Predictions  to  Trial  8  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-3b.  HPAC  and  NARAC  Predictions  to  Trial  8  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-4a.  HPAC  and  NARAC  Predictions  to  Trial  9  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-4b.  HPAC  and  NARAC  Predictions  to  Trial  9  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-5a.  HPAC  and  NARAC  Predictions  to  Trial  10  on  Linear  Scale: 

Stability  Category  is  1 
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Figure  E-5b.  HPAC  and  NARAC  Predictions  to  Trial  10  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-6a.  HPAC  and  NARAC  Predictions  to  Trial  11  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-6b.  HPAC  and  NARAC  Predictions  to  Trial  11  on  Logarithmic  Scale: 

Stability  Category  is  3 


E-13 


DOSAGE 


ARC50 


ARC 100 


20  40  60  80 


Sampler  number 


ARC 4 00 


20  40  60  80 

Sampler  number 

ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgra99_tr<aeer_Experiment_1 2.txt 

PG  Prediction  File  1:  HPAC\nodeposition\p<a_12_novd.out 
PG  Prediction  File  2:  ARAC\nadepositian\pq_1 2 jnovd  a rac 


Figure  E-7a.  HPAC  and  NARAC  Predictions  to  Trial  12  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-7b.  HPAC  and  NARAC  Predictions  to  Trial  12  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-8a.  HPAC  and  NARAC  Predictions  to  Trial  13  on  Linear  Scale: 

Stability  Category  is  7 
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Figure  E-8b.  HPAC  and  NARAC  Predictions  to  Trial  13  on  Logarithmic  Scale: 

Stability  Category  is  7 
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Figure  E-9a.  HPAC  and  NARAC  Predictions  to  Trial  14  on  Linear  Scale: 

Stability  Category  is  7 
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Figure  E-9b.  HPAC  and  NARAC  Predictions  to  Trial  14  on  Logarithmic  Scale: 

Stability  Category  is  7 
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Figure  E-10a.  HPAC  and  NARAC  Predictions  to  Trial  15  on  Linear  Scale: 

Stability  Category  is  1 
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Figure  E-IOb.  HPAC  and  NARAC  Predictions  to  Trial  15  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-11a.  HPAC  and  NARAC  Predictions  to  Trial  16  on  Linear  Scale 

Stability  Category  is  1 
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Figure  E-llb.  HPAC  and  NARAC  Predictions  to  Trial  16  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-12a.  HPAC  and  NARAC  Predictions  to  Trial  17  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-12b.  HPAC  and  NARAC  Predictions  to  Trial  17  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-13a.  HPAC  and  NARAC  Predictions  to  Trial  18  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-13b.  HPAC  and  NARAC  Predictions  to  Trial  18  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-14a.  HPAC  and  NARAC  Predictions  to  Trial  19  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-14b.  HPAC  and  NARAC  Predictions  to  Trial  19  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-15a.  HPAC  and  NARAC  Predictions  to  Trial  20  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-15b.  HPAC  and  NARAC  Predictions  to  Trial  20  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-16a.  HPAC  and  NARAC  Predictions  to  Trial  21  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-16b.  HPAC  and  NARAC  Predictions  to  Trial  21  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-17a.  HPAC  and  NARAC  Predictions  to  Trial  22  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-17b.  HPAC  and  NARAC  Predictions  to  Trial  22  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-18a.  HPAC  and  NARAC  Predictions  to  Trial  23  on  Linear  Scale: 

Stability  Category  is  4 


E-36 


DOSAGE  DOSAGE  DOSAGE 


ARC50 


PG  Prediction  1  to 
PG  Trial  File: 

PG  Prediction  File 
PG  Prediction  File 


Prediction2  Comparison 

pr jg  ra  S9_tr<aee  r_£xperim  e  n  t_23.  t  xt 
1 :  HPAC\nodeposition\pg_23_novd.out 
2:  ARAC\nadepositian\pg _23 jnovd  a rac 


Figure  E-18b.  HPAC  and  NARAC  Predictions  to  Trial  23  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-19a.  HPAC  and  NARAC  Predictions  to  Trial  24  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-19b.  HPAC  and  NARAC  Predictions  to  Trial  24  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-20a.  HPAC  and  NARAC  Predictions  to  Trial  25  on  Linear  Scale: 

Stability  Category  is  1 
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Figure  E-20b.  HPAC  and  NARAC  Predictions  to  Trial  25  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-21a.  HPAC  and  NARAC  Predictions  to  Trial  26  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-21b.  HPAC  and  NARAC  Predictions  to  Trial  26  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-22a.  HPAC  and  NARAC  Predictions  to  Trial  27  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-22b.  HPAC  and  NARAC  Predictions  to  Trial  27  on  Logarithmic  Scale: 

Stability  Category  is  2 


E-45 


DOSAGE 


ARC50 


20  40  60  80 


Sampler  number 


ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_tr<aeer_Experiment_28.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_2a_novd.out 
PG  Prediction  File  2:  ARAC\nadeposition\pg_28.jnovd  arac 


Figure  E-23a.  HPAC  and  NARAC  Predictions  to  Trial  28  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-23b.  HPAC  and  NARAC  Predictions  to  Trial  28  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-24a.  HPAC  and  NARAC  Predictions  to  Trial  32  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-24b.  HPAC  and  NARAC  Predictions  to  Trial  32  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-25a.  HPAC  and  NARAC  Predictions  to  Trial  33  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-25b.  HPAC  and  NARAC  Predictions  to  Trial  33  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-26a.  HPAC  and  NARAC  Predictions  to  Trial  34  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-26b.  HPAC  and  NARAC  Predictions  to  Trial  34  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-27a.  HPAC  and  NARAC  Predictions  to  Trial  35  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-27b.  HPAC  and  NARAC  Predictions  to  Trial  35  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-28a.  HPAC  and  NARAC  Predictions  to  Trial  36  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-28b.  HPAC  and  NARAC  Predictions  to  Trial  36  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-29a.  HPAC  and  NARAC  Predictions  to  Trial  37  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-29b.  HPAC  and  NARAC  Predictions  to  Trial  37  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-30a.  HPAC  and  NARAC  Predictions  to  Trial  38  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-30b.  HPAC  and  NARAC  Predictions  to  Trial  38  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-31a.  HPAC  and  NARAC  Predictions  to  Trial  39  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-31b.  HPAC  and  NARAC  Predictions  to  Trial  39  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-32a.  HPAC  and  NARAC  Predictions  to  Trial  40  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-32b.  HPAC  and  NARAC  Predictions  to  Trial  40  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-33a.  HPAC  and  NARAC  Predictions  to  Trial  41  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-33b.  HPAC  and  NARAC  Predictions  to  Trial  41  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-34a.  HPAC  and  NARAC  Predictions  to  Trial  42  on  Linear  Scale: 

Stability  Category  is  4 


E-68 


30VS0Q  30VS0Q  30VS0Q 


ARC50 


PG  Prediction  1  to 
PG  Trial  File: 

PG  Prediction  File 
PG  Prediction  File 


Prediction2  Comparison 

pr-jgra99_traeer_Experinient_42.txt 
1 :  HPAC\nodeposition\pq_42_novd.out 
2:  ARAC\nadepositian\pg_42_jnovd  a rac 


Figure  E-34b.  HPAC  and  NARAC  Predictions  to  Trial  42  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-35a.  HPAC  and  NARAC  Predictions  to  Trial  43  on  Linear  Scale: 

Stability  Category  is  1 
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Figure  E-35b.  HPAC  and  NARAC  Predictions  to  Trial  43  on  Logarithmic  Scale: 

Stability  Category  is  1 
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Figure  E-36a.  HPAC  and  NARAC  Predictions  to  Trial  44  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-36b.  HPAC  and  NARAC  Predictions  to  Trial  44  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-37a.  HPAC  and  NARAC  Predictions  to  Trial  45  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-37b.  HPAC  and  NARAC  Predictions  to  Trial  45  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-38a.  HPAC  and  NARAC  Predictions  to  Trial  46  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-38b.  HPAC  and  NARAC  Predictions  to  Trial  46  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-39a.  HPAC  and  NARAC  Predictions  to  Trial  48  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-39b.  HPAC  and  NARAC  Predictions  to  Trial  48  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-40a.  HPAC  and  NARAC  Predictions  to  Trial  49  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-40b.  HPAC  and  NARAC  Predictions  to  Trial  49  on  Logarithmic  Scale: 

Stability  Category  is  2 


E-81 


DOSAGE  DOSAGE 


ARC50 


20  40  60  80 

Sampler  number 

ARC 100 


20  40  60  80 

Sampler  number 

ARC  2  00 


20  40  60  80 

Sampler  number 

ARC 4 00 


_ _ * _ _ 

20  40  60  80 


Sampler  number 

ARC 800 


50  100  150 

Sampler  number 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_50.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_50_novd.out 
PG  Prediction  File  2:  ARAC\nadepositian\pq_5Q_jnovd  arac 


Figure  E-41a.  HPAC  and  NARAC  Predictions  to  Trial  50  on  Linear  Scale: 

Stability  Category  is  2 
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Figure  E-41b.  HPAC  and  NARAC  Predictions  to  Trial  50  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-42a.  HPAC  and  NARAC  Predictions  to  Trial  51  on  Linear  Scale: 

Stability  Category  is  2 


E-84 


DOSAGE 


ARC50 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experinient_5l.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pq_51_novd.out 
PG  Prediction  File  2:  ARAC\nadepositian\pg_5l  _/iovd  a  rac 


Figure  E-42b.  HPAC  and  NARAC  Predictions  to  Trial  51  on  Logarithmic  Scale: 

Stability  Category  is  2 
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Figure  E-43a.  HPAC  and  NARAC  Predictions  to  Trial  54  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-43b.  HPAC  and  NARAC  Predictions  to  Trial  54  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-44a.  HPAC  and  NARAC  Predictions  to  Trial  55  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-44b.  HPAC  and  NARAC  Predictions  to  Trial  55  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-45a.  HPAC  and  NARAC  Predictions  to  Trial  56  on  Linear  Scale: 

Stability  Category  is  4 
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Figure  E-45b.  HPAC  and  NARAC  Predictions  to  Trial  56  on  Logarithmic  Scale: 

Stability  Category  is  4 
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Figure  E-46a.  HPAC  and  NARAC  Predictions  to  Trial  57  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-46b.  HPAC  and  NARAC  Predictions  to  Trial  57  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-47a.  HPAC  and  NARAC  Predictions  to  Trial  58  on  Linear  Scale: 

Stability  Category  is  6 
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Figure  E-47b.  HPAC  and  NARAC  Predictions  to  Trial  58  on  Logarithmic  Scale: 

Stability  Category  is  6 
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Figure  E-48a.  HPAC  and  NARAC  Predictions  to  Trial  59  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-48b.  HPAC  and  NARAC  Predictions  to  Trial  59  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-49a.  HPAC  and  NARAC  Predictions  to  Trial  60  on  Linear  Scale: 

Stability  Category  is  5 
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Figure  E-49b.  HPAC  and  NARAC  Predictions  to  Trial  60  on  Logarithmic  Scale: 

Stability  Category  is  5 
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Figure  E-50a.  HPAC  and  NARAC  Predictions  to  Trial  61  on  Linear  Scale: 

Stability  Category  is  3 
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Figure  E-50b.  HPAC  and  NARAC  Predictions  to  Trial  61  on  Logarithmic  Scale: 

Stability  Category  is  3 
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Figure  E-51a.  HPAC  and  NARAC  Predictions  to  Trial  62  on  Linear  Scale: 

Stability  Category  is  2 


E-102 


30VS0Q  30VS0Q  30VS0Q 


ARC50 


20  40  60  80 


Sampler  number 


50  100  150 

Sampler  number 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_62.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pq_62_novd.out 
PG  Prediction  File  2:  ARAC\nadepositian\pg_£S2_jnovd  arac 


Figure  E-51b.  HPAC  and  NARAC  Predictions  to  Trial  62  on  Logarithmic  Scale: 

Stability  Category  is  2 
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APPENDIX  F 

COMPARISONS  OF  HPAC  PREDICTIONS:  WITH  AND  WITHOUT 

SURFACE  DEPOSITION 


This  appendix  presents  graphical  comparisons  of  HPAC  predictions  with  and 
without  the  inclusion  of  a  mechanism  for  SO2  surface  deposition.  Vertical  plot  units  are 
dosage  units  of  mg-sec/m  .  The  horizontal  plot  units  are  sampler  numbers  as  presented  in 
the  Prairie  Grass  field  trials  [Ref.  F- 1] .  Sampler  number  1  is  oriented  to  the  west,  the 
middle  sampler  (45  or  90)  is  oriented  to  the  north,  and  the  last  sampler  (91  or  181)  is 
oriented  to  the  east  of  the  SO2  gas  release  source.  Only  data  values  greater  than  the  cutoff 

•2  a 

threshold  of  3  mg-sec/m  (0.005  mg/m  )  are  presented.  This  cutoff  threshold  value 
corresponds  to  a  minimum  value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  HPAC  predictions  with  and  without  SO2  surface  deposition  are 
presented  on  a  linear  dosage  scale.  Each  graphical  comparison  consists  of  five  plots  (one 
for  each  arc)  with  the  top  plot  depicting  the  50-meter  arc,  the  second  plot  depicting  the 
100-meter  arc,  the  third  plot  depicting  the  200-meter  arc,  and  so  on.  The  last  panel  (just 
above  the  figure  caption)  contains  information  about  the  data  files  used  to  produce  these 
plots.  The  Prairie  Grass  field  trial  file  name  contains  a  two-digit  number  corresponding 
to  the  trial  number.  Prediction  file  names  contain  the  moniker  “nodeposition,”  denoting 
that  SO2  surface  deposition  was  not  considered,  or  the  moniker  “deposition,”  denoting 
that  SO2  surface  deposition  was  considered,  and  a  one-  or  two-digit  number  reflecting  the 
Prairie  Grass  field  trial  being  predicted. 

The  meanings  of  the  colors  used  in  the  plots  are  described  below: 

.  Turquoise  —  dosages  that  overlap  for  both  predictions 


Brown 

— 

higher  dosages  for  predictions  without  SO2  surface 

deposition. 

The  dashed  lines  depict  dosage  values  actually  obtained  in  the  field  trial.  Irwin 
stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure  captions. 
These  stability  category  assignments  are  based  on  Reference  F-2. 
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Figure  F-1.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  5: 

Stability  Category  is  2 
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Figure  F-2.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  7: 

Stability  Category  is  1 
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Figure  F-3.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  8: 

Stability  Category  is  2 
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Figure  F-4.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  9: 

Stability  Category  is  2 
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Figure  F-5.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  10: 

Stability  Category  is  1 
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Figure  F-6.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  11: 

Stability  Category  is  3 
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Figure  F-7.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  12: 

Stability  Category  is  3 
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Figure  F-8.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  13: 

Stability  Category  is  7 
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Figure  F-9.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  14: 

Stability  Category  is  7 
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Figure  F-10.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  15: 

Stability  Category  is  1 
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Figure  F-11.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  16: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_tr<aeer_Experiment_1 7.txt 
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Figure  F-12.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  17: 

Stability  Category  is  5 
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Figure  F-13.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  18: 

Stability  Category  is  5 
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Figure  F-14.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  19: 

Stability  Category  is  2 
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Figure  F-15.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  20: 

Stability  Category  is  3 
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Figure  F-16.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  21 : 

Stability  Category  is  4 
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Figure  F-17.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  22: 

Stability  Category  is  4 
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Figure  F-18.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  23: 

Stability  Category  is  4 
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Figure  F-19.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  24: 

Stability  Category  is  4 
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Figure  F-20.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  25: 

Stability  Category  is  1 
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Figure  F-21.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  26: 

Stability  Category  is  2 
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Figure  F-22.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  27: 

Stability  Category  is  2 
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Figure  F-23.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  28: 

Stability  Category  is  5 
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Figure  F-24.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  32: 

Stability  Category  is  6 
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Figure  F-25.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  33: 

Stability  Category  is  3 
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Figure  F-26.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  34: 

Stability  Category  is  3 
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Figure  F-27.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  35: 

Stability  Category  is  6 
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Figure  F-28.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  36: 

Stability  Category  is  6 
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Figure  F-29.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  37: 

Stability  Category  is  4 
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Figure  F-30.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  38: 

Stability  Category  is  4 
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Figure  F-31.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  39: 

Stability  Category  is  6 
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Figure  F-32.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  40: 

Stability  Category  is  6 
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Figure  F-33.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  41 : 

Stability  Category  is  5 
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Figure  F-34.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  42: 

Stability  Category  is  4 
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Figure  F-35.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  43: 

Stability  Category  is  1 
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Figure  F-36.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  44: 

Stability  Category  is  2 
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Figure  F-37.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  45: 

Stability  Category  is  3 
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Figure  F-38.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  46: 

Stability  Category  is  4 
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Figure  F-39.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  48: 

Stability  Category  is  3 
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Figure  F-40.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  49: 

Stability  Category  is  2 
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Figure  F-41.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  50: 

Stability  Category  is  2 
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Figure  F-42.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  51 : 

Stability  Category  is  2 
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Figure  F-43.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  54: 

Stability  Category  is  5 


F-44 


DOSAGE  DOSAGE 


ARC50 


ARC 100 


20  40  60  80 

Sampler  number 

ARC  2  00 


ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_traeer_Experiment_55.txt 

PG  Prediction  File  1 :  HPAC\deposition\pq_55_vd.out 
PG  Prediction  File  2:  HPAC\nodeposition\pq_55_novd.out 


Figure  F-44.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  55: 

Stability  Category  is  4 
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Figure  F-45.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  56: 

Stability  Category  is  4 
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Figure  F-46.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  57: 

Stability  Category  is  3 
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Figure  F-47.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  58: 

Stability  Category  is  6 
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Figure  F-48.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  59: 

Stability  Category  is  5 
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Figure  F-49.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  60: 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_ExperimenLj6l.txt 

PG  Prediction  File  1:  HPAC\deposition\pq_61_vd.out 
PG  Prediction  File  2:  HPAC\nodeposib‘on\pg_i6l  _novd.out 


Figure  F-50.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  61 : 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgras9_tracer_Experiment_62.txt 

PG  Prediction  File  1 :  HPAC\deposition\pq_62_vd.out 
PG  Prediction  File  2:  HPAC\nodeposition\pq_£2_novd.out 


Figure  F-51.  HPAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  62: 

Stability  Category  is  2 
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APPENDIX  G 

COMPARISONS  OF  HPAC  PREDICTIONS:  WITH  DIFFERENT 
VALUES  OF  DISPERSION  DURING  THE  LIGHTEST  WIND 

CONDITIONS 


This  appendix  presents  graphical  comparisons  of  HPAC  predictions  completed 
with  the  dispersion  parameter  uu(calm)  left  at  the  default  value  of  0.25  and  with  uu(calm) 
set  to  zero.  Vertical  plot  units  are  dosage  units  of  mg-sec/m  .  Horizontal  plot  units  are 
sampler  numbers  as  presented  in  the  Prairie  Grass  field  trials  [Ref.  G-l].  Sampler 
number  1  is  oriented  to  the  west,  the  middle  sampler  (45  or  90)  is  oriented  to  the  north, 
and  the  last  sampler  (91  or  181)  is  oriented  to  the  east  of  the  SOi  gas  release  source. 
Only  data  values  greater  than  the  cutoff  threshold  of  3  mg-sec/m  (0.005  mg/m  )  are 
presented.  This  cutoff  threshold  value  corresponds  to  a  minimum  value  reported  in 
Prairie  Grass  field  trials. 

Comparisons  of  these  predictions  are  presented  on  a  linear  dosage  scale.  Each 
graphical  comparison  consists  of  five  plots  (one  for  each  arc)  with  the  top  plot  depicting 
the  50-meter  arc,  the  second  plot  depicting  the  100-meter  arc,  the  third  plot  depicting  the 
200-meter  arc,  and  so  on.  The  last  panel  (just  above  the  figure  caption)  contains 
information  about  the  data  files  used  to  produce  these  plots.  The  Prairie  Grass  field  trial 
file  name  contains  a  two-digit  number  corresponding  to  the  trial  number.  The  prediction 
file  name  contains  the  moniker  “nodeposition,”  denoting  that  SO2  surface  deposition  was 
not  considered  and  that  uu(calm)  was  set  to  zero.  The  prediction  file  name  containing  the 
moniker  “novd_uucalm_0250”  implies  that  uu(calm)  was  set  to  0.250  (HPAC  default 
value)  and  that  surface  deposition  was  not  considered.  The  one-  or  two-digit  number  in 
the  file  name  reflects  the  Prairie  Grass  field  trial  being  predicted. 

The  meanings  of  the  colors  used  in  the  plots  are  described  below: 


HPAC  predictions  with  dosages  based  on  uu(calm)  set  to 
zero  that  are  higher  than  HPAC  dosages  with  uu(calm)  set 


.  Turquoise  —  HPAC  dosages  that  overlap 


G-l 


.  Brown 


HP  AC  predictions  with  dosages  based  on  uu(calm)  set  to 
0.250  that  are  higher  than  HPAC  dosages  with  uu(calm)  set 
to  0.0. 


The  dashed  lines  depict  dosage  values  actually  obtained  in  the  field  trial.  Irwin 
stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure  captions. 
These  stability  category  assignments  are  based  on  Reference  G-2. 


G-2 


DOSAGE  DOSAGE 


ARC50 


ARC 100 


ARC  2  00 


ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_05.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_5_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_5_navd.out 


Figure  G-1.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  5: 

Stability  Category  is  2 


G-3 


DOSAGE  DOSAGE 


ARC50 


ARC 100 


ARC  2  00 


ARC 4 00 


ARC 800 


50  100  150 

Sampler  number 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_07.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_7_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-j0250\pg_7_navd.out 


Figure  G-2.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  7: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_08.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pgJB_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_£250\pg_8_navd.out 


Figure  G-3.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  8: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr^g  ras9_traeer_ExperimenL_09.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pqJ3_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_j0250\pg_9_navd.out 


Figure  G-4.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  9: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_l0.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_10_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-j025O\pg_l0_/iovd.out 


Figure  G-5.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  10: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_1 1.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_1 1_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D250\pg_l  l_novd.out 


Figure  G-6.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  11 : 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_1 2.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_12_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D25Q\pg_l2_/iovd.out 


Figure  G-7.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  12: 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_l3.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_13_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D250\pg_l  J_novd.out 


Figure  G-8.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  13: 

Stability  Category  is  7 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_1 4.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_14_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D250\pg_l  +_novd.out 


Figure  G-9.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  14: 

Stability  Category  is  7 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_l5.txt 

PG  Prediction  File  1:  HPAC\nodeposition\p<a_15_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D250\pg_l5_novd.out 


Figure  G-10.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  15: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_l6.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_16_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-j025O\pg_l6_/iovd.out 


Figure  G-11.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  16: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr_gras9_tr<acer_Experiment_1 7.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_17_novd.out 
PG  Prediction  File  2:  HPAC\navd-uucalm_025G\pg_l  7_novd.out 


Figure  G-12.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  17: 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_l8.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_ia_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D250\pg_l8_/iovd.out 


Figure  G-13.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  18: 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_l9.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_19_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-j025Q\pg_l9_novd.out 


Figure  G-14.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  19: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_20.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_20_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm.J)25O\pg_20_/iovd.out 


Figure  G-15.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  20: 

Stability  Category  is  3 


G-17 


DOSAGE  DOSAGE 


ARC50 


ARC  2  00 


ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_2l.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_21_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_2l_/iovd.out 


Figure  G-16.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  21 : 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_22.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pq_22_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_22_/iovd.out 


Figure  G-17.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  22: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_23.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_23_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_2J_novd.out 


Figure  G-18.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  23: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_24.txt 

PG  Prediction  File  1:  HPAC\nodeposition\p<a_24_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_2+_/iovd.out 


Figure  G-19.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  24: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_25.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pq_25_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_25_/iovd.out 


Figure  G-20.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  25: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_26.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_26_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm.J)250\pg_26_/iovd.out 


Figure  G-21.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  26: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_27.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_27_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_27_/iovd.out 


Figure  G-22.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  27: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_28.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_28_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm.J)250\pg_28_/iovd.out 


Figure  G-23.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  28: 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_32.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg-32_novd.out 
PG  Prediction  File  2:  HPAC\navd_uucalm_025G\pg_32_novd.aut 


Figure  G-24.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  32: 

Stability  Category  is  6 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_33.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg-33_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-I)250\pg_3J_/iovd.out 


Figure  G-25.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  33: 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_34.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg— 34_novd.out 
PG  Prediction  File  2:  HPAC\navd_uucalm_025G\pg_3+_novd.aut 


Figure  G-26.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  34: 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_35.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pgw35_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_35_novd.out 


Figure  G-27.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  35: 

Stability  Category  is  6 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_36.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pgw36_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-D25Q\pg_36_/iovd.out 


Figure  G-28.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  36: 

Stability  Category  is  6 
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pr-jgra99_traeer_ExperirTient_37.txt 
1 :  HPAC\nodeposition\pg_37_novd.out 
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Figure  G-29.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  37: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_38.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg-38_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm.J)250\pg_38_/iovd.out 


Figure  G-30.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  38: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_39.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_39_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm^D25G\pg_39_novd.out 


Figure  G-31.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  39: 

Stability  Category  is  6 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_40.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_40_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D25O\pg_40_/iovd.out 


Figure  G-32.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  40: 

Stability  Category  is  6 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_41.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pg_41_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_41_novd.out 


Figure  G-33.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  41 : 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_42.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_42_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_£250\pg_42_/iovd.out 


Figure  G-34.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  42: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr_gra99_tracer_Experiment_43.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_43_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_£250\pg_4J_/iovd.out 


Figure  G-35.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  43: 

Stability  Category  is  1 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_44.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_44_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_44_/iovd.out 


Figure  G-36.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  44: 

Stability  Category  is  2 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr_gra99_tracer_Experiment_45.txt 

PG  Prediction  File  1:  HPAC\nodeposition\p<a_45_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_£250\pg_45_/iovd.out 


Figure  G-37.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  45: 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgra99_traeer_Experiment_46.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_46_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_46_/iovd.out 


Figure  G-38.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  46: 

Stability  Category  is  4 
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PG  Trial  File:  pr.jgra99_tr<aeer_Experiment_4S.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_48_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_48_/iovd.out 


Figure  G-39.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  48: 

Stability  Category  is  3 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_49.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_49_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_49_novd.out 


Figure  G-40.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  49: 

Stability  Category  is  2 
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PG  Prediction  File  1 :  HPAC\nodeposition\pg_50_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm-j0250\pg_5Q_/iovd.out 


Figure  G-41.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  50: 

Stability  Category  is  2 
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PG  Prediction  File  1:  HPAC\nodeposition\pg_51_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm.J)250\pg_5l_novd.out 


Figure  G-42.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  51 : 

Stability  Category  is  2 
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pr^g  ra99_traeer_ExperirTient_54.txt 
1 :  HPAC\nodeposition\pg_j54_novd.out 
2:  HPAC\navdjjucalm_D250\pg_5+_/iovd.aut 


Figure  G-43.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  54: 

Stability  Category  is  5 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_55.txt 

PG  Prediction  File  1:  HPAC\nodeposition\pq_J55_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_55_novd.out 


Figure  G-44.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  55: 

Stability  Category  is  4 
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PG  Prediction  File  1 :  HPAC\nodeposition\pg_56_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_j0250\pg_56_novd.out 


Figure  G-45.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  56: 

Stability  Category  is  4 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_57.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_57_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_57_novd.out 


Figure  G-46.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  57: 

Stability  Category  is  3 
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1 :  HPAC\nodeposition\pq_58_novd.out 
2:  HPAC\navd_Liucalrn_0250\pg_58_/iovd.out 


Figure  G-47.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  58: 

Stability  Category  is  6 
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PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr-jgras9_traeer_Experiment_59.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_59_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_59_novd.out 


Figure  G-48.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  59: 

Stability  Category  is  5 
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PG  Trial  File:  pr-jgra99_traeer_Experiment_60.txt 

PG  Prediction  File  1 :  HPAC\nodeposition\pg_60_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_£250\pg_6Q_novd.out 


Figure  G-49.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  60: 

Stability  Category  is  5 
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PG  Prediction  File  1:  HPAC\nodeposition\pg_61_novd.out 
PG  Prediction  File  2:  HPAC\navdjjucalm_D250\pg_6l_novd.out 


Figure  G-50.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  61 : 

Stability  Category  is  3 
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PG  Prediction  File  1 :  HPAC\nodeposition\pg_62_novd.out 
PG  Prediction  File  2:  HPAC\navd_uucalm^025G\pg_62_novd.aut 


Figure  G-51.  HPAC  With  uu(calm)  Set  to  0.0  or  0.250  Predictions  to  Trial  62: 

Stability  Category  is  2 
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APPENDIX  H 

RESULTS  AND  ANALYSES: 
SUPPLEMENTAL  FIGURES  FOR  MOEs 


This  appendix  contains  some  of  the  MOE  comparisons  that  were  done  for  HP  AC 
and  NARAC.  The  first  four  figures  (H-l  through  H-4)  present  MOE  1  comparisons  for 
Cfn  =  Cfp  =  E  The  next  four  figures  (H-5  through  H-8)  present  MOE  1  comparisons  for 
Cfn  =  5  and  Cfp  =  0.5.  MOEs  based  on  AE1  and  AE2  are  presented,  and  comparisons 
based  on  arc  range  and  stability  category  grouping  (SCG)  are  shown.  The  figures  shown 
are  based  on  the  nominal  linear  comparisons  of  predictions  and  observations.  Although 
not  deposited  here,  we  also  computed  all  MOE  1  values  based  the  logarithms  of  the 
compared  predictions  and  observations  (see  the  figures  labeled  with  the  “b”  in 
Appendices  C  and  D). 

The  next  eight  figures  (H-9  through  H-16)  present  MOE  1  (for  Cfn  =  Cfp  =  1  and 
Cfn  =  5,  Cfp  =  0.5)  results  based  on  using  two  probit  curves  as  a  lethality/effects  filter. 
We  computed  these  MOEs  for  the  LD50  values  and  probit  slopes  shown  in  Table  H-l. 
The  example  lethality/effects  filter  figures  shown  in  this  appendix  are  for  LD50  values  of 
100  and  5,000  mg- sec/m3. 

Table  H-1.  LD50  Values  and  Probit  Slopes  Used  for  Lethality/Effects  Filter  Calculations 


LD50  Value 

(mg-sec/m3) 

Slope 

3 

2 

10 

2 

100 

6 

1,000 

6 

2,000 

12 

5,000 

12 

10,000 

12 

50,000 

6 

H-l 


The  comparisons  shown  in  this  appendix  are  based  on  a  threshold  dosage  value  of 
60  mg-sec/m  as  described  in  Chapter  2.  We  also  completed  a  set  of  HPAC  and  NARAC 
comparisons  to  Prairie  Grass  trials  at  a  dosage  threshold  of  3  mg-sec/m3.  The  results  and 
conclusions  based  on  the  two  different  threshold  dosage  values  are  substantially  similar. 
A  few  of  the  results  at  the  lower  threshold  level  are  briefly  described  in  Chapter  3. 

The  next  eight  figures  present  the  comparative  results  for  MOE  2,  the  2D  MOE. 
The  first  four  (H-17  through  H-20)  figures  present  the  results  for  HPAC  and  NARAC 
comparisons  using  AE1  and  AE2.  Comparisons  as  a  function  of  range  and  stability 
category  grouping  are  provided.  The  figures  shown  are  based  on  the  nominal  linear 
comparisons  of  predictions  and  observations.  Although  not  deposited  here,  we  also 
computed  all  MOE  2  values  based  the  logarithms  of  the  compared  predictions  and 
observations  (see  the  figures  labeled  with  the  “b”  in  Appendices  C  and  D). 

The  final  four  figures  (H-21  through  H-24)  present  MOE  2  results  based  on  the 
usage  of  the  lethality/effects  filter  methodology  -  in  this  case,  a  probit  curve.  As  was  true 
for  MOE  1,  the  example  lethality/effects  filter  figures  shown  in  this  appendix  are  for 
LD50  values  of  100  and  5,000  mg-sec/m  .  However,  we  computed  MOE  2  for  all  of  the 
probit  curves  described  in  Figure  H-l. 

All  of  the  MOE  calculations  (with  threshold  =  60  mg-sec/m  )  discussed  in  this 
appendix  were  also  completed  for  comparisons  of  HPAC  predictions  with  and  without 
surface  deposition  and  with  two  settings  of  the  uu(calm)  feature  (0.0  and  0.25). 1 
Similarly,  MOEs  were  computed  for  NARAC  predictions  with  and  without  surface 
deposition  and  included  using  an  experimental  sigma-v  value.  The  figures  associated 
with  the  HPAC  and  NARAC  excursion  comparisons  are  available,  but,  other  than  a  few 
used  directly  in  the  discussions  of  Chapter  3,  they  have  not  been  included  in  this  report. 


1  See  Appendices  F,  G,  L,  and  M  for  additional  information  and  individual  trial  comparisons  under  these 
conditions. 
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Figure  H-1.  MOE  1  (CFn  =  CFP  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {All  Trials  and  By  Range  for  AE1  and  AE2}2 


-  The  “All  Trials”  comparisons  show  two  estimates  of  the  uncertainty  for  each  model.  This  redundancy 
allows  one  to  assess  the  relative  amount  of  variance  due  to  the  choice  of  bootstrap  sample  size,  1,000. 
The  “51”  label  on  the  x-axis  refers  to  the  number  of  Prairie  Grass  trials  considered  in  the  comparison. 


H-3 


Figure  H-2.  MOE  1  (CFn  =  CFp  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials 
{By  Stability  Category  Grouping  (SCG)  for  AE1  and  AE2} 
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Figure  H-3.  MOE  1  (CFn  =  CFP  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials 

{SCG  x  Range  for  AE1} 
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Figure  H-4.  MOE  1  (CFn  =  CFp  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials 

{SCG  x  Range  for  AE2} 
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Figure  H-5.  MOE  1  (CFn  =  5,  CFP  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {All  Trials  and  By  Range  for  AE1  and  AE2}3 


3  The  “All  Trials”  comparisons  show  two  estimates  of  the  uncertainty  for  each  model.  This  allows  one 
to  assess  the  relative  amount  of  variance  due  to  the  choice  of  bootstrap  sample  size,  1,000. 


H-7 


1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


MOE  1  (AE1):  By  Stability  Category  Grouping 


*  f 


►  Mean 

►  50%LCL 

►  50%UCL 

•  80%LCL 

•  80%UCL 
:  95%LCL 
:  95%UCL 


1,2,3  3,4,5  4  5,6,7 

HP  AC,  Stability  Category 


MOE  1  (AE2):  By  Stability  Category  Grouping) 


1 

0.9 

0.8 

0.7 

=  0.6 

CO 

>  0.5 

LLI 

o  0.4 
0.3 
0.2 
0.1 


*  * 


1,2,3  3,4,5  4  5,6,7  6,7 

HPAC,  Stability  Category 


♦  Mean 

♦  50%LCL 

♦  50%UCL 

♦  80%LCL 

♦  80%UCL 
x  95%LCL 
x  95%UCL 


1 

0.9 


0.2 

0.1 


MOE  1  (AE1):  By  Stability  Category  Grouping 


MOE  1  (AE2):  By  Stability  Category  Grouping) 


±± 


♦  Mean 

♦  50%LCL 

♦  50%UCL 

♦  80%LCL 

♦  80%UCL 
x  95%LCL 
x  95%UCL 


1,2,3  3,4,5  4  5,6,7  6,7 

NARAC,  Stability  Category 


1 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 

0.3 

0.2 

0.1 

0 


♦  * 


*  ♦ 


1,2,3  3,4,5  4  5,6,7 

NARAC,  Stability  Category 


>  Mean 

>  50%LCL 

>  50%UCL 
•  80%LCL 

■  80%UCL 

■  95%LCL 
'  95%UCL 


Figure  H-6.  MOE  1  (CFn  =  5,  CFP  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {By  Stability  Category  Grouping  (SCG)  for  AE1  and  AE2} 
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Figure  H-7.  MOE  1  (CFn  =  5,  CFp  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 

Trials  {SCG  x  Range  for  AE1} 
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Figure  H-8.  MOE  1  (CFN  =  5,  CFP  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 

Trials  {SCG  x  Range  for  AE2} 
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Figure  H-9.  MOE  1  (CFN  =  CFP  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {All  Trials  and  By  Range  for  Two  Probit  Curves}4 


4  The  “All  Trials”  comparisons  show  two  estimates  of  the  uncertainty  for  each  model.  This  allows  one 
to  assess  the  relative  amount  of  variance  due  to  the  choice  of  bootstrap  sample  size,  1,000. 
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Figure  H-10.  MOE  1  (CFN  =  CFp  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {By  Stability  Category  Grouping  (SCG)  for  Two  Probit  Curves} 
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Figure  H-11.  MOE  1  (CFn  =  CFP  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {SCG  x  Range  for  LD50  =100,  Probit  Slope  =  6} 
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Figure  H-12.  MOE  1  (CFn  =  CFp  =  1.0):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {SCG  x  Range  for  LD50  =5,000,  Probit  Slope  =  12} 
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MOE  1  (LD50  =  5,000,  Slope  =  12):  By  Range  (in  meters) 
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MOE  1  (LD50  =  100,  Slope  =  6):  By  Range  (in  meters) 
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Figure  H-13.  MOE  1  (CFn  =  5,  CFp  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {All  Trials  and  By  Range  for  Two  Probit  Curves}5 


5  The  “All  Trials”  comparisons  show  two  estimates  of  the  uncertainty  for  each  model.  This  allows  one 
to  assess  the  relative  amount  of  variance  due  to  the  choice  of  bootstrap  sample  size,  1,000. 


H-15 


1 

0.9 

0.8 

0.7 

<D 

B  0.6 

ra 

>  0.5 

LJ 

O  0.4 
0.3 
0.2 
0.1 
0 


MOE 1  (LD50  =  100,  Slope  =6):  By  SCG 


t  < 


♦  Mean 

♦  50%LCL 

♦  50%UCL 
.  80%La 

♦  80%ucl 

X  95%La 
X  95%ua 


1,2,3  3,4,5  4  5,6,7 

HPAC  Stability  Category 


6,7 


MOE  1  (LD50  =  5,000,  Slope  =  12):  By  SCG 


1 

0.9 

0.8 

0.7 

<D 

3  0.6 

TO 

>  0.5 

LJ 

O  0.4 
0.3 
0.2 
0.1 
0 


*  t 


♦  IVfean 

♦  50%LCL 

♦  50%UCL 

♦  80%LCL 

♦  80%UCL 
-  95%LCL 
x  95%UCL 


1,2  1,2,3  3,4,5  4  5,6,7  6,7 

HPAC  Stability  Category 


1 

0.9 

0.8 

07 

(1) 

B  0.6 

(0 

>  0.5 

LJ 

O  0.4 
0.3 
0.2 
0.1 
0 


MOE  1  (LD50  =  100,  Slope  =6):  By  SCG 


♦  IVfean 

♦  50%LCL 

♦  50%UCL 

♦  80%LCL 

♦  80%UCL 

x  95%LCL 

x  95%UCL 

i  ♦  ♦ 

*  ♦  *  T 

!  T  ♦ 

X 

1,2  1,2,3  3,4,5  4  5,6,7 

NARAC  Stability  Category 


6,7 


MOE  1  (LD50  =  5,000,  Slope  =  12):  By  SCG 


1 

0.9 

0.8 

0.7 

a) 

3  0.6 

(0 

>  0.5 

LJ 

O  0.4 
0.3 
0.2 
0.1 
0 


-+ 


i  l  ? 


♦  » 


♦  IVfean 

♦  50%LCL 

♦  50%UCL 

♦  80°/cLCL 

♦  80%UCL 
x  95%LCL 
x  95%UCL 


1,2  1,2,3  3,4,5  4  5,6,7  6,7 

NARAQ  Stability  Category 


Figure  H-14.  MOE  1  (CFn  =  5,  CFp  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {By  Stability  Category  Grouping  (SCG)  for  Two  Probit  Curves} 
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Figure  H-15.  MOE  1  (CFN  =  5,  CFp  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {SCG  x  Range  for  LD50  =  100,  Probit  Slope  =  6} 
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Figure  H-16.  MOE  1  (CFn  =  5,  CFp  =  0.5):  HPAC  and  NARAC  Comparisons  to  Prairie  Grass 
Trials  {SCG  x  Range  for  LD50  =  5,000,  Probit  Slope  =  12} 
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Figure  H-17.  MOE  2:  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials  {All  Trials  and 

By  Range  for  AE1  and  AE2}6 


6  The  points  shown  in  the  above  plots  correspond  to  the  approximate  95th  percent  confidence  regions 
associated  with  the  MOE  2  point  estimates.  These  confidence  regions  were  obtained  via  bootstrap 
techniques.  The  actual  point  estimates  lie  at  approximately  the  center  of  these  confidence  regions. 
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Figure  H-18.  MOE  2:  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials  {By  SCG  for 

AE1  and  AE2}7 


7  Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5; 
6,7  and  1,2,3;  4;  5,6,7.  The  points  shown  in  the  above  plots  correspond  to  the  approximate  95th  percent 
confidence  regions  associated  with  the  MOE  2  point  estimates.  These  confidence  regions  were 
obtained  via  bootstrap  techniques.  The  actual  point  estimates  lie  at  approximately  the  center  of  these 
confidence  regions. 
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Figure  H-19.  MOE  2  Point  Estimates  Only:  HPAC  and  NARAC  Comparisons  to  Prairie 

Grass  Trials  {SCG  x  Range  for  AE1}8 


8  Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5 
6,7  and  1,2,3;  4;  5,6,7. 
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Figure  H-20.  MOE  2  Point  Estimates  Only:  HPAC  and  NARAC  Comparisons  to  Prairie 

Grass  Trials  {SCG  x  Range  for  AE2}9 


9  Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5 
6,7  and  1,2,3;  4;  5,6,7. 
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Figure  H-21.  MOE  2:  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials  {All  Trials  and 

By  Range  for  Two  Probit  Curves}10 


The  points  shown  in  the  above  plots  correspond  to  the  approximate  95th  percent  confidence  regions 
associated  with  the  MOE  2  point  estimates.  These  regions  were  obtained  via  bootstrap  techniques. 
The  actual  point  estimates  lie  at  approximately  the  center  of  these  confidence  regions. 
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Figure  H-22.  MOE  2:  HPAC  and  NARAC  Comparisons  to  Prairie  Grass  Trials  {By  SCG  for 

Two  Probit  Curves}11 


Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5; 
6,7  and  1,2,3;  4;  5,6,7.  The  points  shown  in  the  above  plots  correspond  to  the  approximate  95th  percent 
confidence  regions  associated  with  the  MOE  2  point  estimates.  These  confidence  regions  were 
obtained  via  bootstrap  techniques.  The  actual  point  estimates  lie  at  approximately  the  center  of  these 
confidence  regions. 
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Figure  H-23.  MOE  2  Point  Estimates  Only:  HPAC  and  NARAC  Comparisons  to  Prairie 
Grass  Trials  {SCG  x  Range  for  LD50  =  100,  Probit  Slope  =  6}12 


Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5 
6,7  and  1,2,3;  4;  5,6,7. 
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Figure  H-24.  MOE  2  Point  Estimates  Only:  HPAC  and  NARAC  Comparisons  to  Prairie 
Grass  Trials  {SCG  x  Range  for  LD50  =  5,000,  Probit  Slope  =  12}13 


I3  Two  different  independent  combinations  of  stability  category  groupings  are  shown  above  -  1,2;  3,4,5 
6,7  and  1,2,3;  4;  5,6,7. 
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APPENDIX  I 

RESULTS  AND  ANALYSES:  SUPPLEMENTAL  FIGURES  AND 
TABLES  FOR  STANDARD  STATISTICAL  MEASURES 


This  appendix  provides  figures  and  tables  that  describe  comparisons  of  HP  AC 
and  NARAC1  predictions  of  Prairie  Grass  trials  based  on  standard  statistical  measures. 
The  statistical  measures  included  in  this  appendix  are  fractional  bias  (FB),  normalized 
mean  square  error  (NMSE),  geometric  mean  bias  (MG),  geometric  mean  variance  (VG), 
correlation  coefficient  (R),  fraction  of  predictions  within  a  factor  of  2  (FAC2),  fraction  of 
predictions  within  a  factor  of  5  (FAC5),  and  fraction  of  predictions  within  a  factor  of  10 
(FAC  10). 

The  first  six  figures  (1-1  through  1-6)  present  the  results  for  computations 
involving  the  ArcMax.  Charts  that  display  FB,  NMSE,  VG,  and  MG  are  provided  for  all 
trials  and  as  a  function  of  range  and  SCG.  The  next  six  figures  (1-7  through  1-12)  provide 
the  results  obtained  for  crosswind  integrated  (CWI)  dosage.  Since  NMSE  and  VG  for  a 
model  can  be  due  to  mean  bias  and  random  error,  there  are  minimum  values  of  NMSE 
and  VG  for  a  given  FB  and  MG,  respectively.  The  limiting  curves  described  on  some  of 
the  charts  represent  the  minimum  possible  values  of  NMSE  and  VG  and  were  computed 
in  the  following  way: 


AFB 

NMSE  = 

(1-1) 

4 -FB- 

ln  VG  =  (in  MGf 

(1-2)2 

Figures  1-13  through  1-15  provide  tables  that  describe  the  values  obtained  for 
FAC2,  FAC5,  and  FAC  10  for  ArcMax  and  CWI  dosage.  In  several  of  the  figures  and 
tables,  “U,”  “N,”  and  “S”  refer  to  stability  category  groupings  unstable,  neutral,  and 
stable,  respectively. 


1  NARAC  is  also  referred  to  as  ARAC  in  the  figures  of  this  appendix. 
-  See  Chapter  2,  Reference  2-9. 
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The  next  three  figures  (1-16  through  1-18)  present  FB  and  NMSE  values  for  the 
plume  width  parameter  computed  via  the  equal  cuts-off-the-sides  method  (Wec).  See 
Appendix  J  for  additional  details  associated  with  this  novel  parameter. 

The  final  three  figures  (1-19  through  1-21)  provide  results  for  the  parameter 
CentDir.  The  statistical  measures  root  mean  square  error  (RMSE)  and  bias  were 
computed  for  this  parameter.  See  Appendix  J  for  additional  discussion. 
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Figure  1-1.  FB  and  NMSE:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass  Trials 

{All  Trials  and  By  Range} 
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Figure  1-2.  MG  and  VG:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass  Trials 

{All  Trials  and  By  Range} 
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Figure  1-3.  FB,  NMSE,  and  R:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass 
Trials  {By  SCG  -  U  =  1,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-4.  MG  and  VG:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass  Trials 

{By  SCG  -  U  =  1,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-5.  FB,  NMSE,  R:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass  Trials 

{By  SCG  -  U  =  1,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-6.  MG  and  VG:  HPAC  and  NARAC  ArcMax  Comparisons  to  Prairie  Grass  Trials 
{By  SCG  -  U  =  1 ,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-7.  FB  and  NMSE:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie  Grass 

Trials  {All  Trials  and  By  Range} 
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Figure  1-8.  MG  and  VG:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie  Grass 

Trials  {All  Trials  and  By  Range} 
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Figure  1-9.  FB,  NMSE,  and  R:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie 
Grass  Trials  {By  SCG  -  U  =  1 ,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-10.  MG  and  VG:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie  Grass 
Trials  {By  SCG  -  U  =  1,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-11. 


FB,  NMSE,  and  R:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie 
Grass  Trials  {By  SCG  -  U  =  1,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-12.  MG  and  VG:  HPAC  and  NARAC  CWI  Dosage  Comparisons  to  Prairie  Grass 
Trials  {By  SCG  -  U  =  1,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-13.  FAC2,  FAC5,  and  FAC10  Tables:  HPAC  and  NARAC  Comparisons  to  Prairie 

Grass  Trials  {All  Trials  and  By  Range} 
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Figure  1-14.  FAC2,  FAC5,  and  FAC10  Tables:  HPAC  and  NARAC  Comparisons  to  Prairie 
Grass  Trials  {By  SCGs  -  U  =  1,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-15.  FAC2,  FAC5,  and  FAC10  Tables:  HPAC  and  NARAC  Comparisons  to  Prairie 
Grass  Trials  {By  SCGs  -  U  =  1,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-16.  FB  and  NMSE:  HPAC  and  NARAC  Wec  Comparisons  to  Prairie  Grass  Trials 

{All  Trials  and  By  Range} 
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Figure  1-17.  FB,  NMSE,  and  R:  HPAC  and  NARAC  Wec  Comparisons  to  Prairie  Grass  Trials 

{By  SCG  -  U  =  1 ,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-18.  FB,  NMSE,  and  R:  HPAC  and  NARAC  Wec  Comparisons  to  Prairie  Grass  Trials 

{By  SCG  -  U  =  1 ,2,3;  N  =  4,;  and  S  =  5,6,7} 
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Figure  1-19.  Bias  and  RMSE:  HPAC  and  NARAC  CentDir  Comparisons  to  Prairie  Grass 

Trials  {All  Trials  and  By  Range} 
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Figure  1-20.  Bias  and  RMSE:  HPAC  and  NARAC  CentDir  Comparisons  to  Prairie  Grass 
Trials  {By  SCG  -  U  =  1,2;  N  =  3,4,5;  and  S  =  6,7} 
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Figure  1-21.  Bias  and  RMSE:  HPAC  and  NARAC  CentDir  Comparisons  to  Prairie  Grass 
Trials  {By  SCG  -  U  =  1,2,3;  N  =  4,;  and  S  =  5,6,7} 
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APPENDIX  J 

SOME  COMMENTS  ON  THE  ESTIMATION  OF  PLUME  WIDTH 

AND  DIRECTION 


There  are  several  ways  that  one  might  define  the  centerline  direction  (CentDir) 
and  width  for  the  observed  dosages  along  a  given  Prairie  Grass  field  trial  arc.  We 
considered  two  methods  of  defining  the  centerline  or  direction  (with  corresponding 
definitions  for  the  widths),  plus  a  separate  method  of  determining  the  “width”  of  the 
observation  that  does  not  involve  the  direction  of  the  plume.  Here,  we  define  the  plume 
in  terms  of  dosage,  not  concentration.  Therefore,  the  plume  widths  are  defined  in  terms 
of  crosswind  integrated  dosage. 


1.  METHOD  I  (CENTER  OF  MASS) 


This  method  describes  the  centerline  of  the  plume,  based  on  the  center  of  mass 
(COM).1  We  define  the  centerline  of  the  plume  at  any  given  arc  by: 


COM  = 


yv  id. 

4—H=\  i 

Y,v  D 

1  ' 


(J-l) 


where  D,  is  a  dosage  collected  at  sampler  i. 

Then,  given  a  particular  fraction  of  the  crosswind  integrated  dosage  that  we  want 
to  capture  inside  the  plume  (e.g.,  0.75),  we  find  the  smallest  offset  k,  such  that: 


v  half  -width 

Then  the  width  of  the  plume  is  2khaifWidth- 


=  mini  k  0.75V  D,  <  yCOM+k  £>,.}. 

L  t—h  i  /—li=COM-k  1  J 


(J-2) 


1  Reference  J-l  uses  this  definition,  with  the  exception  that  the  center  of  mass  of  the  plume 
concentration  is  considered. 
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2.  METHOD  II  (MEDIAN  OF  MASS) 

This  method  is  based  on  defining  the  centerline  as  a  point  that  divides  the  plume 
into  two  equally  weighted  plumes  (i.e.,  this  is  a  median-based  calculation).  We  define 
the  median  of  mass  (MOM)  centerline  of  the  plume  for  any  given  arc  by  finding  the 
sampler  with  index,  MOM,  such  that: 

MOM  =  minj  k  |  D,  >  0.5£",  D, },  (J-3) 

where  D,  is  a  dosage  collected  at  sampler  i. 

Then,  given  a  particular  fraction  of  crosswind  integrated  dosage  that  we  want  to 
capture  inside  the  plume  (e.g.,  0.75),  we  find  the  smallest  offset  k  such  that: 

***—  =  mi"l*  |  0-751, D<  S  TZtA-  <J-4> 

Then  the  width  of  the  plume  is  2khaif-width- 


3.  METHOD  III  (EQUAL  CUTS  OFF  THE  SIDES): 


This  method  defines  the  width  of  the  plume  independent  of  the  centerline  (or 
plume  direction).  For  this  method,  one  “cuts”  equal  crosswind  integrated  dosages  from 
the  sides  of  the  plume.  Given  a  particular  fraction  of  the  crosswind  integrated  dosage  that 
we  want  to  capture  inside  the  plume  (e.g.,  0.75),  we  can  find  the  maximal  index  kieft,  and 
minimal  index  krigilt  (from  the  left  and  right,  respectively)  such  that: 


Kft  =  maxj  k  X;=iZ>  <  1-<p  =  0.125 1 
knght  =  minj  k  s  |^  =  0.12sJ 


(1-5) 


Then,  we  define  width  for  this  equal-cuts-off-the-sides  method  (Wec)  as: 

Wec  =rdAknght-kleft)  (J'6) 


where  r  is  the  distance  in  meters  to  the  arc  and  0  is  the  angular  separation  (in  radians) 
between  the  samplers.  Wec  is  estimated  in  units  of  meters. 

Figures  J-l  and  J-2  demonstrate  the  above  definitions  for  two  Prairie  Gras  field 
trials.  The  blue  lines  denote  calculations  based  on  the  COM  method,  the  yellow  lines 
denote  calculations  based  on  the  MOM  method,  and  the  black  lines  denote  calculations 
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based  on  the  “equal-cuts-off-the-sides”  method.  For  the  majority  of  the  Prairie  Grass 
trials,  these  different  definitions  produced  very  similar  results. 

Figure  J-l  corresponds  to  a  case  where  there  is  a  discernible  disagreement 
between  the  methodologies.  It  can  be  seen  that  the  observations  associated  with  this  field 
trial  were  not  particularly  Gaussian-like  (nor  were  they  even  symmetric).  Figure  J-2 
corresponds  to  a  case  somewhat  more  typical  of  the  Prairie  Grass  field  trials.  In  this 
case,  there  is  very  good  agreement  among  the  different  methodologies.  The  results 
presented  in  this  appendix  and  in  Chapter  3  are  based  on  the  MOM  method  to  compute 
the  plume  centerline  direction  (CentDir)  and  the  equal-cuts-off-the-sides  method  to 
estimate  the  plume’s  width  (Wec). 
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Figure  J-1.  Centerline  and  Width  Estimation,  “Worst  Case”: 

Blue  -  COM-Based,  Yellow  -  MOM-Based,  Black  -  Equal  Areas  Cut  Off  the  Sides-Based 
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Figure  J-1.  Centerline  and  Width  Estimation,  “Best  Case”: 

Blue  -  COM-Based,  Yellow  -  MOM-Based,  Black  -  Equal  Areas  Cut  Off  the  Sides-Based 
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PRAIRIE  GRASS  FIELD  TRIAL  DATA  ANOMALIES 
THAT  WERE  CORRECTED 


A  computer-ready  version  of  Prairie  Grass  field  trial  observations  was  provided 
by  Lawrence  Livermore  National  Laboratory  [Ref.  K-l].  Later,  we  discovered  that 
hardcopy  of  the  field  trial  data  were  presented  in  the  original  Prairie  Grass  report  [Ref. 
K-2].  During  our  analysis  of  the  Prairie  Grass  field  trial  data,  we  encountered  two  types 
of  data  anomalies:  “spurious”  peaks  reported  at  some  samplers1  and  apparent  data 
transcription  mistakes  that  occurred  when  data  were  copied  from  the  hardcopy  of  the 
report  into  the  computer  file.  Examination  of  the  field  trial  data  revealed  that  “spurious” 
peaks  most  likely  corresponded  to  misplacement  of  the  decimal  point  by  the  authors  of 
the  Prairie  Grass  report  [Ref.  K-2].  After  consultation  with  a  researcher  who  has 
previously  used  Prairie  Grass  field  trial  data  [Ref.  K-3],  we  decided  to  correct  the 
“spurious”  peaks  by  fixing  the  decimal  point  misplacements  (using  our  processing 
software). 

The  majority  of  computer  data  transcription  errors  occurred  as  a  result  of  incorrect 
placement  of  the  decimal  point.  Only  field  trials  that  were  used  in  the  “official” 
HPAC/NARAC  comparison  were  scanned  for  data  transcription  mistakes.  At  present,  no 
tower  sampler  data  were  checked  since  they  were  not  used  in  our  HPAC/NARAC 
comparisons. 

Tables  K-l,  K-2,  and  K-3,  that  follow,  provide  additional  details. 


1  We  somewhat  arbitrarily  define  a  data  peak  to  be  spurious  if  it  is  more  than  an  order  of  magnitude 
larger  than  the  next  largest  value  in  the  arc  under  consideration. 
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Table  K-1.  List  of  Prairie  Grass  Trials  That  Were  Scanned  for  Transcription  Errors 


(5 


,  7,  8,  9,  10,  11,  12,  13,  14,  15,  16,  17,  18,  19,  20,  21, 22,  23,  24,  25,  26,  27,  28,  32,  33,  34,  35, 
36,  37,  38,  39,  40,  41 , 42,  43,  44,  45,  46,  48,  49,  50,  51 ,  54,  55,  56,  57,  58,  59,  60,  61 , 62} 


Table  K-2.  Prairie  Grass  Trials  That  Were  Modified  as  a  Result  of  Decimal  Point  and 
Missing  Value  Data  Transcription  Errors 


Trial  #,  arc,  sampler  # 

Old  value 

New  Value 

Comments 

T7,  800m,  SI  02 

0.004 

0.037 

factor  of  1 0 

T7,  800m,  SI  03 

0.092 

0.009 

T8,  50m,  S44 

949.400 

94.940 

factor  of  0.1 

Til,  800m,  S90 

missing 

408.500 

“spurious”  peak  added 

T13,  50m,  S65 

0.728 

0.073 

factor  of  0.1 

T21, 400m,  S40 

0.105 

1.055 

factor  of  1 0 

T23,  50m,  S22 

899.70 

89.965 

factor  of  0.1 

T45,  800m,  S77 

0.132 

1.320 

factor  of  1 0 

T51 ,  100m,  S67 

missing 

6.717 

appears  valid 

T51, 50m,  S80 

missing 

210.24 

appears  valid 

Table  K-3.  Prairie  Grass  Trials  That  Have  “Spurious”  Peaks  Replaced  by  Fixing  Decimal 

Point  Position  (Done  in  Software  Only) 


Trial  #,  arc,  sampler  # 

Old  Value 

New  Values 

Comments 

Til,  800m,  S90 

408.500 

0.408 

factor  of  250  greater  than  next  largest  value 

T16,  200m,  S63 

329.000 

3.29 

factor  of  55  greater  than  next  largest  value 
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WITHOUT  SURFACE  DEPOSITION 


This  appendix  presents  graphical  comparisons  of  NARAC  predictions  with  and 
without  the  inclusion  of  a  mechanism  for  SO2  surface  deposition.  Vertical  plot  units  are 

a 

dosage  units  of  mg-sec/m  .  The  horizontal  plot  units  are  sampler  numbers  as  presented  in 
the  Prairie  Grass  field  trials  [Ref.  L-l].  Sampler  number  1  oriented  to  the  west,  the 
middle  sampler  (45  or  90)  oriented  to  the  north,  and  the  last  sampler  (91  or  181)  is 
oriented  to  the  east  of  the  SO2  gas  release  source.  Only  data  values  greater  than  the  cutoff 

•2  a 

threshold  of  3  mg-sec/m  (0.005  mg/m  )  are  presented.  This  cutoff  threshold  value 
corresponds  to  a  minimum  value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  NARAC  predictions  with  and  without  SO2  surface  deposition  are 
presented  on  linear  and  log  dosage  scales.  Each  graphical  comparison  consists  of  five 
plots  (one  for  each  arc)  with  the  top  plot  depicting  the  50-meter  arc,  the  second  plot 
depicting  the  100-meter  arc,  the  third  plot  depicting  the  200- meter  arc,  and  so  on.  The 
last  panel  (just  above  the  figure  caption)  contains  information  about  the  data  files  used  to 
produce  these  plots.  The  Prairie  Grass  field  trial  file  name  contains  a  two-digit  number 
corresponding  to  the  trial  number.  Prediction  file  names  contain  the  moniker  “novd” 
denoting  that  SO2  surface  deposition  was  not  considered,  or  the  moniker  “vd”  denoting 
that  SO2  surface  deposition  was  considered;  and  a  one-  or  two-digit  number  reflecting  the 
Prairie  Grass  field  trial  being  predicted.  Even-numbered  pages  contain  figures  on  the 
linear  dosage  scale  while  odd-numbered  pages  contain  figures  on  the  logarithmic  dosage 
scale. 

The  meanings  of  the  colors  used  in  the  plots  are  described  in  the  below  legend: 


Dark  Blue  — 

Hhigher  dosages  for  predictions  with  S02  surface  deposition 

Turquoise  — 

dosages  that  agree  for  both  predictions. 

Brown  — 

higher  dosages  for  predictions  without  SO2  deposition. 

The  dashed  lines  depict  dosage  values  actually  obtained  in  the  field  trial.  Irwin 
stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure  captions. 
These  stability  category  assignments  are  based  on  Reference  L-2. 
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Figure  L-1a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  5  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-lb.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  5  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-2a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  7  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-2b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  7  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-3a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  8  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-3b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  8  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-4a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  9  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-4b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  9  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-5a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  10  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-5b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  10  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-6a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  11  on 

Linear  Scale:  Stability  Category  is  3 


L-12 


ARC50 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_tr<aeer_Experiment_1 1.txt 

PG  Prediction  File  1 :  ARAC\sv_vd\pq_1 1_sv_vd.aroc 
PG  Prediction  File  2:  ARAC\sv_navd\pq_l 1  _sv_novd.orac 


Figure  L-6b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  11  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-7a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  12  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-7b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  12  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-8a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  13  on 

Linear  Scale:  Stability  Category  is  7 
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Figure  L-8b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  13  on 

Logarithmic  Scale:  Stability  Category  is  7 
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Figure  L-9a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  14  on 

Linear  Scale:  Stability  Category  is  7 
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Figure  L-9b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  14  on 

Logarithmic  Scale:  Stability  Category  is  7 
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Figure  L-10a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  15  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-IOb.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  15  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-11a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  16  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-llb.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  16  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-12a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  17  on 

Linear  Scale:  Stability  Category  is  5 
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Figure  L-12b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  17  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-13a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  18  on 

Linear  Scale:  Stability  Category  is  5 
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Figure  L-13b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  18  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-14a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  19  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-14b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  19  on 

Logarithmic  Scale:  Stability  Category  is  2 


L-29 


DOSAGE  DOSAGE 


ARC50 


ARC 100 


ARC  2  00 


ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_traeer_Experiment_20.txt 

PG  Prediction  File  1 :  ARAO0v_vd\pq_2O_sv_vd.oroc 
PG  Prediction  File  2:  ARAC\sv_navd\pq_2Q_sv_n  ovd. arac 


Figure  L-15a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  20  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-15b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  20  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-16a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  21  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-16b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  21  on 

Logarithmic  Scale:  Stability  Category  is  4 
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Figure  L-17a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  22  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-17b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  22  on 

Logarithmic  Scale:  Stability  Category  is  4 
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Figure  L-18a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  23  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-18b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  23  on 

Logarithmic  Scale:  Stability  Category  is  4 
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Figure  L-19a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  24  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-19b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  24  on 

Logarithmic  Scale:  Stability  Category  is  4 
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Figure  L-20a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  25  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-20b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  25  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-21a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  26  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-21b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  26  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-22a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  27  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-22b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  27  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-23a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  28  on 

Linear  Scale:  Stability  Category  is  5 
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Figure  L-23b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  28  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-24a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  32  on 

Linear  Scale:  Stability  Category  is  6 
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Figure  L-24b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  32  on 

Logarithmic  Scale:  Stability  Category  is  6 


L-49 


30VS0Q  30VS0Q 


ARC50 


20  40  60  80 

Sampler  number 

ARC 100 


20 


40 

Sampler  number 

ARC  2  00 


60 


80 


10000  E- 
8000  E- 
6000  E- 
4000  E- 
2000  =- 


20  40  60  80 


Sampler  number 
ARC 4 00 


ARC 800 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_traeer_Experiment_33.txt 

PG  Prediction  File  1 :  ARACs3v_vd\pq_J3_sv_vd.aroc 
PG  Prediction  File  2:  ARAC\sv_navd\pq_33_sv_n  ovd. arac 


Figure  L-25a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  33  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-25b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  33  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-26a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  34  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-26b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  34  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-27a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  35  on 

Linear  Scale:  Stability  Category  is  6 
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Figure  L-27b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  35  on 

Logarithmic  Scale:  Stability  Category  is  6 
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Figure  L-28a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  36  on 

Linear  Scale:  Stability  Category  is  6 
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Figure  L-28b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  36  on 

Logarithmic  Scale:  Stability  Category  is  6 
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Figure  L-29a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  37  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-29b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  37  on 

Logarithmic  Scale:  Stability  Category  is  4 
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Figure  L-30a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  38  on 

Linear  Scale:  Stability  Category  is  4 
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Figure  L-30b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  38  on 
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Figure  L-31b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  39  on 

Logarithmic  Scale:  Stability  Category  is  6 
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Figure  L-32a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  40  on 

Linear  Scale:  Stability  Category  is  6 
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Figure  L-33a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  41  on 
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Figure  L-33b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  41  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-34b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  42  on 
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Figure  L-35a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  43  on 

Linear  Scale:  Stability  Category  is  1 
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Figure  L-35b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  43  on 

Logarithmic  Scale:  Stability  Category  is  1 
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Figure  L-36a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  44  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-36b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  44  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-37a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  45  on 
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Figure  L-37b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  45  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-38a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  46  on 

Linear  Scale:  Stability  Category  is  4 


L-76 


DOSAGE  DOSAGE  DOSAGE  DOSAGE 


ARC50 


20  40  60  80 


20  40  60  80 

Sampler  number 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  prjgra99_traeer_Experiment_46.txt 

PG  Prediction  File  1 :  ARAC\sv_vd\pq_46._sv_vd.aroc 
PG  Prediction  File  2:  ARAC\sv_navd\pq_46_sv_n  ovd. arac 


Figure  L-38b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  46  on 
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Figure  L-39a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  48  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-39b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  48  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-40a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  49  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-40b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  49  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-41a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  50  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-41b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  50  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-42b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  51  on 

Logarithmic  Scale:  Stability  Category  is  2 
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Figure  L-43b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  54  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-45b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  56  on 
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Figure  L-47b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  58  on 
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Figure  L-48b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  59  on 
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Figure  L-49b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  60  on 

Logarithmic  Scale:  Stability  Category  is  5 
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Figure  L-50a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  61  on 

Linear  Scale:  Stability  Category  is  3 
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Figure  L-50b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  61  on 

Logarithmic  Scale:  Stability  Category  is  3 
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Figure  L-51a.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  62  on 

Linear  Scale:  Stability  Category  is  2 
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Figure  L-51b.  NARAC  With  and  Without  S02  Surface  Deposition  Predictions  to  Trial  62  on 

Logarithmic  Scale:  Stability  Category  is  2 
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APPENDIX  M 

COMPARISONS  OF  NARAC  PREDICTIONS:  WITH  DIFFERENT 
VALUES  OF  DISPERSION  (SIGMA-V  EXPERIMENTAL  AND 

SIGMA-V  CALCULATED) 


APPENDIX  M 

COMPARISONS  OF  NARAC  PREDICTIONS:  WITH  DIFFERENT 
VALUES  OF  DISPERSION  (SIGMA-v  EXPERIMENTAL  AND 

SIGMA-v  CALCULATED) 


This  appendix  presents  graphical  comparisons  of  NARAC  predictions  using  two 
different  sigma-v  values  -  an  experimentally  determined  value  and  a  calculated  value. 
Vertical  plot  units  are  dosage  units  of  mg-sec/m  .  The  horizontal  plot  units  are  sampler 
numbers  as  presented  in  the  Prairie  Grass  field  trials  [Ref.  M-l].  Sampler  number  1 
oriented  to  the  west,  the  middle  sampler  (45  or  90)  oriented  to  the  north,  and  the  last 
sampler  (91  or  181)  is  oriented  to  the  east  of  the  SO2  gas  release  source.  Only  data  values 

o  a 

greater  than  the  cutoff  threshold  of  3  mg-sec/m  (0.005  mg/m  )  are  presented.  This  cutoff 
threshold  value  corresponds  to  a  minimum  value  reported  in  Prairie  Grass  field  trials. 

Comparisons  of  NARAC  predictions  are  presented  on  linear  and  log  dosage 
scales.  Each  graphical  comparison  consist  of  five  plots  (one  for  each  arc)  with  the  top 
plot  depicting  the  50-meter  arc,  the  second  plot  depicting  the  100-meter  arc,  the  third  plot 
depicting  the  200-meter  arc,  and  so  on.  The  last  panel  (just  above  the  figure  caption) 
contains  information  about  the  data  files  used  to  produce  these  plots.  The  Prairie  Grass 
field  trial  file  name  contains  a  two-digit  number  corresponding  to  the  trial  number. 
Prediction  file  names  contain  the  moniker  “novd”  denoting  that  SO2  surface  deposition 
was  not  considered  and  a  calculated  value  for  sigma-v  was  used.  The  moniker  “sv_novd” 
denotes  that  SO2  surface  deposition  was  not  considered  and  an  experimentally 
determined  value  for  sigma-v  was  used.  A  one-  or  two-digit  number  describes  the 
Prairie  Grass  field  trial  being  predicted.  Odd-numbered  pages  contain  figures  on  the 
linear  dosage  scale  while  even-numbered  pages  contain  figures  on  the  logarithmic  dosage 
scale. 

The  meanings  of  the  colors  used  in  the  plots  are  described  in  the  below  legend: 


|higher  dosages  for  predictions  with  calculated  sigma-v. 


.  Turquoise  —  dosages  that  agree  for  both  predictions. 

.  Brown  —  higher  dosages  for  predictions  with  experimental  sigma-v. 


M-l 


The  dashed  lines  depict  dosage  values  actually  obtained  in  the  field  trial.  Irwin 
stability  categories  that  were  used  in  our  analyses  are  denoted  in  the  figure  captions. 
These  stability  category  assignments  are  based  on  Ref.  M-2. 
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Figure  M-1a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  5 

on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-lb.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  5 
on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-2a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  7 

on  Linear  Scale:  Stability  Category  is  1 


M-5 


DOSAGE  DOSAGE 


ARC50 


ARC  2  00 


ARC 4 00 


PG  Prediction  1  to  Prediction2  Comparison 
PG  Trial  File:  pr.jgras9_tr<aeer_Experiment_07.txt 

PG  Prediction  File  1 :  ARACV>odeposition\p<a_7_novd.oroc 
PG  Prediction  File  2:  ARAC\sv_navd\pq_7_sv_navd.arac 


Figure  M-2b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  7 
on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  M-3a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  8 

on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-3b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  8 
on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-4a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  9 

on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-4b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  9 
on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-5a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  10 

on  Linear  Scale:  Stability  Category  is  1 
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Figure  M-5b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  10 

on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  M-6a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  11 

on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-6b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  11 

on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-7a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  12 

on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-7b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  12 

on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-8a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  13 

on  Linear  Scale:  Stability  Category  is  7 
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Figure  M-8b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  13 

on  Logarithmic  Scale:  Stability  Category  is  7 
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Figure  M-9a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  14 

on  Linear  Scale:  Stability  Category  is  7 
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Figure  M-9b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial  14 

on  Logarithmic  Scale:  Stability  Category  is  7 
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Figure  M-10a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

15  on  Linear  Scale:  Stability  Category  is  1 
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Figure  M-IOb.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
15  on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  M-11a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

16  on  Linear  Scale:  Stability  Category  isl 
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Figure  M-llb.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
16  on  Logarithmic  Scale:  Stability  Category  isl 
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Figure  M-12a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

17  on  Linear  Scale:  Stability  Category  is  5 
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Figure  M-12b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
17  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  M-13a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

18  on  Linear  Scale:  Stability  Category  is  5 
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Figure  M-13b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
18  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  M-14a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

19  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-14b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
19  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-34b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
42  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  M-35a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

43  on  Linear  Scale:  Stability  Category  is  1 
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Figure  M-35b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
43  on  Logarithmic  Scale:  Stability  Category  is  1 
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Figure  M-36a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

44  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-36b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
44  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-37a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

45  on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-37b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
45  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-38a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

46  on  Linear  Scale:  Stability  Category  is  4 
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Figure  M-38b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
46  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  M-39a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

48  on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-39b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
48  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-40a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

49  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-40b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
49  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-41a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

50  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-41b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
50  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-42a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

51  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-42b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
51  on  Logarithmic  Scale:  Stability  Category  is  2 
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Figure  M-43a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

54  on  Linear  Scale:  Stability  Category  is  5 
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Figure  M-43b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
54  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  M-44a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

55  on  Linear  Scale:  Stability  Category  is  4 
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Figure  M-44b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
55  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  M-45a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

56  on  Linear  Scale:  Stability  Category  is  4 
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Figure  M-45b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
56  on  Logarithmic  Scale:  Stability  Category  is  4 
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Figure  M-46a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

57  on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-46b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
57  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-47a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

58  on  Linear  Scale:  Stability  Category  is  6 
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Figure  M-47b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
58  on  Logarithmic  Scale:  Stability  Category  is  6 
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Figure  M-48a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

59  on  Linear  Scale:  Stability  Category  is  5 
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Figure  M-48b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
59  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  M-49a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

60  on  Linear  Scale:  Stability  Category  is  5 
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Figure  M-49b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
60  on  Logarithmic  Scale:  Stability  Category  is  5 
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Figure  M-50a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

61  on  Linear  Scale:  Stability  Category  is  3 
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Figure  M-50b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
61  on  Logarithmic  Scale:  Stability  Category  is  3 
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Figure  M-51a.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 

62  on  Linear  Scale:  Stability  Category  is  2 
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Figure  M-51b.  NARAC  Sigma-v  Experimental  and  Sigma-v  Calculated  Predictions  to  Trial 
62  on  Logarithmic  Scale:  Stability  Category  is  2 
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DC-9-1797 


TITLE:  Support  for  DTRA  and  LLNL  in  the  Validation  Analysis  of  Hazardous 
Material  Transport  and  Dispersion  Prediction  Models 

This  task  order  is  for  work  to  be  performed  by  the  Institute  for  Defense  Analyses 
(IDA)  under  Contract  DASW01-98-C-0067,  for  the  Defense  Threat  Reduction  Agency 
(DTRA)  in  the  Department  of  Defense.  This  task  order  authorizes  funding  for  FY  2000. 


1.  BACKGROUND: 

The  Hazard  Prediction  and  Assessment  Capability  (HP AC)  is  a  suite  of  codes  that 
predicts  the  effects  of  hazardous  material  releases  into  the  atmosphere  and  their  impact 
on  civilian  and  military  populations.  The  software  can  use  integrated  source  terms,  high- 
resolution  weather  forecasts,  and  particulate  transport  models  to  predict  hazard  areas 
produced  by  battlefield  or  terrorist  use  of  weapons  of  mass  destruction  (WMD),  by 
conventional  counterforce  attacks  against  WMD  facilities,  or  by  military  and  industrial 
accidents.  HP  AC  is  a  forward  deployable,  counterproliferation  and  counterforce 
capability  software  tool  available  for  government,  government-related,  or  academic  use. 
This  tool  assists  warfighters  in  selecting  weapon  mixes  for  targets  containing  WMD  and 
in  emergency  response  to  hazardous  agent  release.  HPAC’s  relatively  fast-running, 
physics-based  algorithms  enable  users  to  model  and  predict  hazard  areas  and  human 
collateral  effects  in  minutes. 

The  DTRA  Verification  and  Validation  (V&V)  Program  represents  ongoing 
activities  performed  in  parallel  with  development  of  all  predictive  codes  in  support 
of  HPAC.  One  element  of  V&V  is  to  perform  code-on-code  comparisons.  In  this 
strategy,  each  code  receives  the  same  input.  In  this  manner,  differences  in  the 
output  predictions  can  lead  to  the  identification  of  software  bugs,  or  help  to  assess 
technical  strengths  and  weaknesses  of  component  algorithms  within  each  code.  In 
addition,  a  certain  amount  of  credibility  for  both  models  is  achieved  when  their 
predictions  agree.  When  the  inputs  are  simple,  such  as  for  fixed  winds  and  simple 
terrain,  the  predictions  tend  to  be  dominated  by  the  dispersion  algorithms. 
Comparisons  at  this  level  of  complexity  are  important  to  establish  fundamental 
dispersion  algorithm  veracity,  and  to  help  discover  software  bugs.  As  more 
complex  terrain  and  weather  is  included  as  input,  the  number  of  physical  processes 
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responsible  for  transport  and  dispersion  increases  and  the  predictions  become  the 
result  of  many  interdependent  algorithm  calculations. 

Code-on-code  comparisons  will  be  performed  using  the  DTRA  code  HP  AC, 
the  Lawrence  Livermore  National  Laboratory  (LLNL)  code  Atmospheric  Release 
Advisory  Capability  (ARAC),  and,  possibly,  Sandia-developed  codes.  These  codes 
represent  major  national  investments  in  transport  and  dispersion  modeling  within 
their  respective  applications.  The  comparisons  will  provide  information  from 
which  to  validate  the  HP  AC  and  ARAC  models  (and  perhaps  others),  as  well  as 
provide  an  opportunity  to  advance  both  technologies.  The  code  comparisons  will 
include  short,  medium,  and  long-range  transport  distances.  Complex  terrain  and 
weather  will  also  be  included. 

It  is  very  difficult  to  separate  meteorological  uncertainty  from  the  transport 
and  dispersion  model  accuracy  when  comparing  predictions  to  field-trial  validation 
quality  or  real-world  data.  The  validation  challenge  is  to  assess  whether  a  model 
performs  well  over  different  field  trials,  and  ultimately  reflects  real-world 
phenomena.  Some  codes  perform  better  under  certain  conditions  and  specific 
scenarios.  Hazard  prediction  models  are  generally  developed  for  a  range  of  user 
communities  and  applications.  Each  user  community  has  a  different  set  of 
requirements.  Thus,  the  corresponding  hazard  models  tend  to  be  optimized  for 
specific  applications.  The  process  of  accrediting  a  model  is  always  couched  in 
terms  of  the  end-user  requirements. 

Various  figures-of-merit  (FOM)  are  used  to  express  model  performance 
relative  to  observed  data.  Most  FOMs  tend  to  use  manifestations  of  a  ratio 
(geometric  or  arithmetic)  between  the  predicted  and  measured  quantities.  The 
compared  quantities  are  usually  peak,  plume-centerline,  and  off-axis  concentration 
or  dosage,  as  well  as  crosswind  and  along- wind  spread  and  area  coverage.  Other 
FOMs  may  include  the  second-moment  of  the  dosage  and  concentration  values  at  a 
sampler  location.  All  these  FOMs  are  reasonable  validation  performance  measures, 
but  none  of  them  explicitly  expresses  an  application-oriented  performance  measure. 
A  “yardstick”  is  needed  that  measures  application-oriented  model  performance. 
The  scale  on  this  yardstick  would  clearly  and  directly  relate  to  specific  user’s 
concerns  and  needs.  The  pursuit  of  this  “accreditation”  performance  measure  is  a 
new  initiative  at  DTRA. 


2.  OBJECTIVE: 

IDA  will  conduct  independent  analysis  and  special  studies  associated  with 
verification  and  validation  of  the  suite  of  models  associated  with  the  Hazard 
Assessment  and  Prediction  Capability.  IDA  will  support  development  of  user- 
oriented  performance  measures  of  effectiveness  (MOE)  using  validation  quality 
field  trial  data  sets;  coordinate  scenario  definition  and  arbitration  for  code-on-code 
V&V  activities;  and  assist  DTRA  and  the  Department  of  Energy  in  identifying  the 
V&V  parameter  space  associated  with  various  hazard  assessment  and  collateral 
effects  communities. 

The  objectives  of  verification  and  validation  analysis  and  coordination  are: 
(1)  to  ensure  that  a  consistent  analysis  approach  is  used  when  comparing  model 
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predictions,  and  assist  DTRA  in  the  implementation  of  code-on-code  analysis, 
comparisons,  and  interpretation;  and  (2)  to  define  measures  of  effectiveness  in 
terms  of  user-specific  objectives  and  applications. 

The  scope  of  this  effort  may  be  expanded  to  other  programs  as  directed  by 

DTRA. 


3.  STATEMENT  OF  WORK: 

As  required  by  DTRA  technical  representatives,  IDA  will  perform  the  following 

tasks: 


a.  Support  the  planning,  implementation,  arbitration,  and  evaluation  of  code-on- 
code  comparison  activities.  The  purpose  of  these  activities  is  to  compare  the  transport 
and  dispersion  algorithms  and  corresponding  output  predictions  between  DTRA’s  suite  of 
models,  the  DOE  ARAC’s  suite  of  transport  and  dispersion  models,  and  other  models  as 
called  for.  IDA  will  support  code-on-code  scenario  definition,  coordinate  the 
identification  and  implementation  of  common  performance  measures,  and  support 
development  of  a  common  analysis  approach.  IDA  will  conduct  independent  analysis,  as 
needed,  to  support  the  code-on-code  analysis  and  interpretation  of  the  results. 

b.  Explore  validation  and  accreditation  MOEs  given  a  framework  that  includes 
quantification  of  false  positive  and  false  negative  predictions.  This  exploration  would 
include  the  computation  of  MOE  values  for  various  formulations  based  on  short-range 
comparisons  of  HPAC  and  ARAC  predictions  to  field  trial  data.  A  key  to  interpreting  the 
results  of  this  effort  will  be  obtaining  a  sense  for  what  are  the  acceptable  user 
requirements.  These  requirements  will  differ  among  potential  user  groups  (military 
targeting,  passive  CB  defense,  civilian  first  responders,  military  versus  civilian 
population  human  effects,  etc.). 


4.  CORE  STATEMENT: 

This  research  is  consistent  with  IDA’s  mission  in  that  it  will  support  specific 
analytical  requirements  of  the  sponsor  and  will  assist  the  sponsor  with  planning  efforts. 
Accomplishment  of  this  task  order  requires  an  organization  with  experience  in 
operationally  oriented  issues  from  a  joint  and  combined  perspective,  which  IDA,  a 
Federally  Funded  Research  and  Development  Center,  is  able  to  provide.  It  draws  upon 
IDA’s  core  competencies  in  Systems  Evaluations  and  Operational  Test  and  Evaluation. 
Performance  of  this  task  order  will  benefit  from  and  contribute  to  the  long-term 
continuity  of  IDA’s  research  program. 
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